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FOREWORD 



The research done for this report by the Human Resources Research Organization is 
part of an overall effort tmder Work Unit SMMART to provide a manual for Anny 
trainers to use in selecting the most cost-effective toaining^ metiiods and^media for specific 
training tasks. .This report concludes. Work Sub-Unit SA^ARTI, the purpose of whidi 
Wa^ to prepare a research plan for otiier SMMART work sub-units; the research plan pas ^ 
prepared separately . > . , 

Research imder Work: Unit^ SMMART* is beixig conducted by HumRRO Division 
No. 2 at Fort Knox» Kentucky. The Division Direcl^t is Dr. Dbnald Haggakd. 

Military support for the resc^urch is being provided by the UJS. Army ;^rmor Human - 
Rij«sarch Unit. The Unit^^^^ Z . . 

This report contains cop^^ted material; pennisdon hus been' obtuned for its use. 

JHumRRO , research for the De{Murtment of the Army^^ is conducted imder Army 
Contract DAHC 19-73<;-0004. Army Trainini Research is performed under Army Project 
2Q062167A745. , ?/ v ; ; : ^ 



Humn Resources Reseaich^^^ 



mi- 



HBaBm_, 



Ml - 




MILITARY PROBLEM 

Systems engineering of Army training programs, as defined in CON ARC 
Reg* 350-100-1,^ provides for Systematic consideration of critical factors in developing a 
program. When skillfully followed^ these procedures replace most of the intuitive 
processes previously used. Still, training systems engineering retains the rulesof-thumb 
usually i^pUed fqr selecting cost-effective training methods and media, and Army schools 
have foimd these to be unsatisfactory. The need, as expressed by the XJJSl Continental 
Army Command (CON ARC is for empirically based criteria and procedures for selecting 
the most cost-effective methods and media for specific training tasks. 



RESEARCH OBJECTIVE 



The objective of Work Sub-Unit SMMART I was to prepare a plan for research 
toward empirical determination of criteria and procedures for optimal selection of 
cost-effective methods and media. To detenpine a realistic basis for the plan,- research 
intb^existijig pertinent k^ t? 



APPROACH \ ^ 

, Ilie procedure foUowed fcr prepiuration of this report was u review of pertinent 
Utemtiire,^ mialysis of the findings in search of useful information, identification of 
problems to be resolved m further research, and fonnulation of new approaches (where 



RESULTS " 

The reselurch results fell into two niajor categories: (a) those pertainiri^ to methods- 
media definition and classification, and methods:media sdectio;r^ procedures, 
and (b) those pe&*taining to traming cost-effectivoiess and ^cdy^aT^roce^ 

The litemture review yielded little of inmiediate value/The empirical data foimd on 
the relative cost-effectiveness of m^^thods and media are msufficient as a basis for reliable 
selection of methods and media for specific training' tasks . Also, the existing methods- 
media selection procedures, training cost'analysis proceduresy^ approaches 
for developing such procedures are inadequate for Army needs, although portions of 
some of these may be useful in developing procedures for Army use* Possible approaches 
for remo^ng these inadequacies are in cl^^^ 



^U.S* Continental Anny Command. Trainittg: Syitemi Engineering of training (Couru Detign). 
^See Appendices F and G. ' *' 
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INTRODUCTION 



BACKGROUND 

Syitems mgineering of Amy tnininf procmns, as defined in CONARC 
Reg. 3S0*100*1»' provides for systematic considention of critical fkctors in developing a 
progranu When skillfulty followed, these procedures refrtace most of the intuitive 
processes previous^ used. Even so» ttafning qfstems engineering still retains the rulerof* 
thumb means usually qnpiied Ibr sdecting cost^ective tiaining meOiods and media, and 
Anny sdiook have found these to be unsatisfsctory^ 

The Human Resources ReseardTOiganisation proposed that rsseardi be undertaken 
to ultimately correct these defidencies throu^ devdopment of empirically based 
sdection criteria and procedurss. this resulted in two CONARC-sponsored Work Units- 
MEDIA and GdStHtt HumRRO Division No. 2. 

The primary objective of Work Unit MEDIA wis to develop a method for improving 
media implraientation to meet specific train&fig objectives in Army training {programs* 
rtot of this work was also concerned with a procedure for ejecting the most satfsfikctory 
training methods* 

The primary objective of Work Unit COST was to evaluate the feasibiUty of a 
preliminsry model for comparing the cost^ffectiveneis of training media* 

Expb>ratory researdi as part of Wwk Unit MEDIA* viMch i^ 
of the literature* showed that neither MttaUe guUelines nor sufficient empirical data were 
avaibble fkpm whidi to derive adequate criteria and procedures for the most satisfsctory 
sslectk>n and use of methods and media. Therefore* in December 1971* after a HumRRO* 
OdNARC conference on VloA Units MEDIA and COST* the efforts of these units vrere 
combhied into an expanded program of research known as Work Unit SMMART— 
/ ^Devdoping Criteria for the Sdection of Methods and Media by Army Tninm. 



PURPOSE AHO SCOPE OF REPORT 

The purpose of this report is to fulfill the first reqiUrement of Work Unit SMMART* 
that is* to summarize the state-of-knowledge pertaining to the sdection of cpst^ffective 
triUiing methods md inedia/Rdevsnt findings i^ the literature are dcicdbed and problems 
are discussed that must be resohred in the dev^pment of reliable procHures for selecting 
the most cost^tfective mettiods«iedia combinations and training programs. Since the report 
reveals many deficiencies that may be alleviated through further research* it mi\st not be 
considered as a manual.for use in selecting training methods and media. 

The report is di^ed into two parts: Part I concerns methods and media selection 
procedures* iriiile Part H has to do with analyses of training cost fkctors. 

Bemuse of the topical nature of the ch4>ter discussions and the extensive use of the 
literature* ttie relevant references are listed at the fnd of eadi duq[>ter. Tae full literature 
review lists* for Parts I and II reqpecttvdy* are pn^mted in the final two ^>pendicet of 
thereporX 

^VS. ContintnU'' Armsr Command* Training: Syiiim$ Engimmring of Training (Coum Demgn), 
ODNARC Ragulatioii 350*100*1, Fori Monroe, Viif ink, Ftbrtuury 19e8» 
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Chapter 1 



EXISTING METHOQS-MEDIA SELECTION PROCEDURES 

Reliable pioceduies for selecting the most satisfactory instructional methods and 
media for qiedfic purposes seem to constitute an^ ultimate goal foremost educational 
researdi» but the research results have yielded only meagor guidelines for use in educa- 
tional planning. 

Despite the lack ofadequate onpurical data on whidi to base deyelc^mental effects, 
several mettiods^iedm selection procedures have been devised in and out of the Army. 
Thoae {Ntocedures deviaed outside are summarized in Appendix A, except for one proce- 
dure tiiat is discussed in tikis duqpte^ 

Eadi of the procedum presented in Appendix A is for the s^ecti<m of both 
inettiods media^ as 9^ procedure murt be in ord«r to be meuui^Efid. Neither 
me^ods nor media* wtei comideied apart, have any practicar meaiiing, even thou^ 
educidors and, trabers often telk and write about the two «sparat^. No modd cim be 
used without some mednmi or media to inclement it. Lftewise, no medium can be used 
wtthout scMne method ^b^g biplementfed. Therefore, it a procedure in Appendix A does 
iK>t qpecifiod^ indicate noMctk 

None of Uie sevmd metiiodsHOMKlia setectioh procedures already in existence satisfies 
Army requirements for the fpDowingTreasoiis: . ^ 

(1) Most selection procediuei devdoped dutdde the Arany vrere designed for 
use in dvilian sdbook, and the others were based tq^n specific resfarictiv^ assunqitions 
a^d theories that do not sufficiently iq[>ply to Army conditions. The sdection procedures 
dtevdoped witii£ai the Amy were based entirdy iq[K>ii the procedures intended for use in 
civilian sdiools» and tiiey have proved to unsatisfactory for Army rituations> The 
reasons for the defideiuaesrare that many Army training requirements, sudi as trouble- 
iriiooting and leMleidiq;>, have no counterparts in civilian educatioii, and Army training 
deals with numerous otiier factors that differ from tiiose in civilian education, such as 
time restrictions, student ages, course content, and traming envuonments; 

' (2) Existing selection procedures used in civUian scfaobls have ptoved 'to be 
unsuccessftil even for thdr inbnided purposes^ One reason for the failure lis that the 
selection criteria used were too gaieral, too gross for use in idratifying spedTic media for 
q>ecific purposes. As Pryludk and Snow(l) steted, instructional media are most effec- 
tive used when unique nuedia atbributes are associated with specific subject matter 
(teadiing points) and stadent leammg activities. None of the existing sdection procedures 
have provided for this. 

b addition to being inadequate, most Costing selection procedures are too conq>li* 
cated, and efforts to amplify them have resulted in over-amplification, whidi created 
even greater inadequades. A practicd selection procedure diould avoid boOi pitfalls: 
ovor-simplification, whidi is inadequate, and over-complication, whidi is too laborious* 
--\ An approadi different from ttiose in Ai^>endix A wi^ introduced by Walker(2). 
Atthotq^ he is not conastent in distinguishing betwem methods and media, and^thouf^ 
his list of ''techniques** is incomplete and indudes some of doubtful value. Walker's 
4>proadi can be usefrd if it is thorouglily and carefully developed* 

Walker first prepared a list of tedimques, vrtiich included lecture, job-experience 
training, <m4hejob training, discusnon sessions, texts, televisim' (TV), programed 



instruction (PI), laboratory work, audiovisual (AV) equipment, simulation, trainers, tests, 
hypnosis, dri^s, and sleep teaching. He then listed the criteria (Table 1) to be used for 
.methods-media selection and prepared a matrix. ^ 



Table 1 

Selection Criteria for Training^ 



Selection Criteria 



Percentage of Technir^l , 
Training Personnel W*)o 
Mentioned These Criteria 



- 1. TIME TO PRODUCE 100 

2. NO. OF CTUDENTS TAUGHT . .... . . 83 

3. EFFECTIVENESS OF TEACHING MOTOR SKILLS y .......... 75 

4. EFFECTIVENESS OF TEACHING THEORY . . . \ . . . . . 75 

5. AMOUNT OF DATA NEEDED TO PREPARE . . : 75 

6. roSTTOTEACM;.^. 75 

7. COST TO DEVELOP ....>...;..!.. .... 75 ' / 

8. FACILITIES NEEDED TO PRESENT .... .............. 67 

9. SPECIFIC OBJECTIVES > . . . : f 67 

10. LEVEL OF STUDENT INTELLIGENCE . 58 

11: STUDENT KNOWLEDGE ............... 58 

12. Sttjdeht Ihfothratipn ........... . ^ ^ 33 

13. QUALtTY OF JSPECIALISI^ TO DEVELOP MATERIAL 33 

14. EASE OF EVALUATING STUDENTS . ^ . 25 

15. FACiUTIES NEEDED TO PREPARE 25 

16. EASE OF ADMINISTRATION . 17 

t7. COMPLEXITY OF THE TRAINING MATERIAL 17 

18. FLEXiBILITY .. 17 

19. NO. OF INSTRUCTORS TO INSTRUCT 17 

20. ' NO. OF INSTRUCTORS TO DEVELOP *. . . . 17 

21. Student Reinforcement 17 

22. Student Participation 17 

•23. Realism 8 

*24. Transfer of Training 8 * 

25. MANEUVERABILITY 8 

26. Variety of psychological/learning proceisses 8 

27. Student feedback . 0 

28. RetentK>n ^ 0 

29. Student-paced ^..^ 0 

30. No. of senses stimulated 0 

31. LENGTH OF COURSE . . . _ 0 . 

*32. Simulation of the operational environment 0 

33. EASE OF PRESENTATION ; . . 0 

34. Competition ^ 0 

CAPITALS « managiementHxntered items 

Lower case * student-centered jteim 

* * equally student- and management*centered items 

*R.W. Walker. "An Evaluation of Training Methods and Their Characteristics/' Hamgn Factors, (2) 
® Human Factors Society. Reprinted by permission. 



To determine the value of each technique in relation to each selection criterion, he 
asked a group of training experts to rate each technique on a s^e of 1 to 5 (1 being the 
lowest value, and 5 the hi^est)*" All selection criteria were assiuned to have the same 
weigtit (to be of equal importance). When the experts completed , their ratings, Walker 
calculate the mean values for each technique, dhown^ih Table 2. The result is a 
technkjuei^ref erence index based upon the judgments of training experts. 



Table 2 

Selection Criteria Matrix: Mean Evaluation Ratings' 
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To use the index for selecting preferred techniques, Walker prepared the following 
three-step selection procedure: 

(1) Select those criteria that are relevant to an instructional task. For example, 

from Table 2 one might select the criteria-~realism, student reinforcement, ^ 
transfer of tiaining, and student motivation. ^ 

(2) Sum the mean rating values for each technique on *^the criteria selected in 
Stepl. 

(3) Choose the technique that has the highest total rating (the largest sum of 
mean ratings as determined in Step 2). 

Although Walker's procedure is' not based upon empirical data, his approach might 
be used to develop a procedure for use by^he Army^until an anpirically based procedure 
can be derived. For Army purposes, better lists of methods and media would have to be 
provided. Also, the metiiods and media ratings assigned by training experts would have to 
be according to selection criteria meaningful to Army trainers. 

The UJS.-Army Southeastern Signsd School (3) took an approach somewhat similar 
to Walker's in an attempt to determine how methods and media are selected for Army 
cou**ses. They sent a list of eight questions to 24 Army schools, and received 18 
responses (one in each of 18 schools) On the basis of those responses and information 
gleaned from the literature, the Souliieaster^ School prepsured 18 principles to 

guide trsuners in selecting methods^^^^ ^ 

Besides not meeting .^rmy requirements for an empirically abased procedure, the 
results of this approach can be questioned on at least two counts: 

(1) The eight questions were too few and too broad to yield enough precise 
information on which to base reliable conclusions, 

(2) An analysis was not made across the various levels of training-sy:tem 
planning or across the various course types. . 

In summary, the Army needs. a new methods-media selection procedure to meet its 
particular requirenients, not because tiiere is a lack of existing procedures, but because 
none of them can be used or be ada))ted for use in selecting the most effective methods 
and media for Army courses. 



1. 
2. 
3. 
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Chapter 2 

^ A NEW APPROACH TO METHODS SELECTION 

Since existing procedures for selecting instructional methods and forme^ approaches 
to the development of sudi procedures have proved inadequate, a new approach is 
needed as the basis for further research* «^ 

The foUowing approach, (K>nsisting of foiur major ta^ 

(1) Identify teaching^oint (a knowledge or skill to be learned) characteristics 
that can be used in selecting training methods. 

(2) Ftmctiondiy define and classify training methods and techniques. 

(3) Derive a means for relating teacfaing^poinit diaracteristics to appropriate 
training methods. / - 

(4) Identify student individual differences tiiat can be used in selecting the 
most satisfactory training methods and techniques. 

- Information pertaining to each of these tasks is presented and discussed in this 
chapter. . ^ . ^ _ ' . - 

^ CHARACTERISTIC OF TEACHING POINTS 

A teaching point is a knowledge or skill to be learned. In Army training qrstems 
engineering / essential knowledges and skills to be learned in a course are stated ^in the 
training objectives and subject matter that are specified in the Training Analysis Informa- 
tion Sheets. Thus, ii^onnation on knowledges and skills is readily available so that the 
characteristics they have that can be used for selecting training methods can be 
determined. 

CHARACTERISTICS USEFUL FOR SELECTION 

The characteristics of teadiing poirits that probably can be most useful for selecting 
methods (and media) ure those that indicate the, processes within the nervous system of 
the student that are required for ileturning. These processes, discussed at length by 
Glaser(D» Gaghe(2), and DeCecco (3), are reidfi'njr (associating), discriminating^ and 
generalizing: 

Relating is establishing an association between particular stimuli and particular 
responses. ^ 

Discriminating is distinguishing between appropriate and imqppropriate stimuli. 

Generalizing is associating the correct response with appropriate but somewhat 
different stimuli. 

All tracbing points require the process of relating if the"* are to be learned. In 
addition, some require discrimination learning to prevent confusion of stimuli so that the^ 
learner can discern differencei, such ks differences in makes and models of urtieeled 
vehicles. Others require generalization learning so that the student can recognize common 



characteristics of classes, such as machine guns, aircraft, ^technical manuals, and other 
classes of military objects. 

These teaching-point learning requirements cannot be used alone in selecting training 
methods, but they can be very useful when a particular kind of relating, discriminating, 
or generalizing is identified; that is, one must note whether the relating, discriminating, or 
generalizing must be visual, aural, tactile, olfactory^ gustatory, or kinesthetic/ 
proprioceptive. These sensory requirements imply/ of cpurse, that the student lecuming 
activities must be seeing, heariiii;^ toudiing/feeling^ smelling, tasting, or muscular coordi* 
nating (as in motor-skill learning). 

Eor use in selecting training methods, these student activities must be translated into 
terms that more commonly denote student activities in a learning situation/ terms that 
can be used to relate student activities to training methods. Thus, seeing should^ be 
represented by either obterving ot reading; hearing should be represented by listening; 
and tou(4iing/feeling, smelling, tasting, and muscular coordinating should be represented 
\>y doing. These four categories of studait activities-observing, reading, listening, 
doingHl^est describe, student learning activities for the puipose of selecting training 
methods. These activities are called the means* by which teaching points can be learned. 

RULES FOR^RELATING TEACHING POINTS TO MEANS OF LEARNING 

Since leaming-byndoing is an essential Anhy training principle, it should lie api^ied 
m practical exercises as a part of eveiy instructional program; wd since doing is always - 
iixlicat.ed as a means of leammg botii mental and motor skills, the practical exercise as a 
tmining me^ is always indicated for these purposes. Thus, identification of this 
method as one to be used^is notdif ficult. 

. The major problem in developing an adequate naethod ^lection^procedure is to 
provkle a reliable means of kientifying those m^^ which students can acquire the 

knowledge that they must apply in a: practical exercise, or the knowledge ttiat will 
prepare thm to practicje in a practicd^ 

In formidating^^^^^ teadiing pointe to their means of learning, we 

considered whether: . 

(1) The teaming poInU are in a imcUcd exercise. 

For example, a practical exerdae to teadi map reading would require a 
map for j^resentlng the tetdiing points; therefore, t map wouki be at least 
^ part of the material in a practiod emcbe. 

(2) The teadiing points lEijijai in a pncUi^ 

- For example, an lUuitratlon to $iA learning of a concept, such at a picture 

of a fly beside the Empire State BuUdlng to convey the concept of the 
relaUonshIp of a micron to an Indi. Once the concept Is learned, sudi an 
4,iUustra*tion'w6ukl not be useful in performing either the imictlcal exercise 
or the job task to which the concept is related. 

(3) The teadiing poInU require relating, discrlnilnating, or generalizing, as 
dlscuned previously. 

Following are the rules we formulated for relating teaching points to their iqppro- 
priate means of leaniing: ^ . . ^ 

Observing. Teadiing points are learned by obftsnring If the' practical exercise or 
teaching point requires: 

(1) Associating visual stimuli with acUons. 

(2) Discriminating between viaual stimuli before or . during the practical^ 
exarclse. 

(3) Oenerslizing to different sUmult before or during the practical exercise. 



(4) Visual Action (e^,, to visually select, match, recognize, identify; to 
assist motor action, sudi as drawing, painting, carving, calibrating an 
ohmmeter, tuning a radio receiver), - ^ 

Reading, Teaching points are learned by reading if the practical exercise or 
teadiing point requires: . 

(1) Associating printed verbal materials with actions, 

(2) Discriminating between printed veil>al materials, 

(3) Generalizing to different inrinted verbal materials (e^,, recognizing that . 
varioiu kinds of printed materials belong to a given class, such as Field 
Manuals),' 

(4) AcUon Jn relation to printed verbid materials (e,g,, writing a staff study 
~ or preparing written orders). 

Listening , Teadiing points are learned by llstmiing If the practical exercise or 
^ teadiing point requires: 

(1) Associating . verbal or iionverbal sounds with action (e^,, spoken 
commancfs, reveille), 

(2) Discrin^naUng between verbal or nonverbal sounds, 

(3) Genenriizing to different verbal or nonverbal sounds (e^., perceiving an 
essential meaning when expressed In different words, recognizing that 
actual night surveillance radar sounds are the same as heard on tape 
reWrders In the dassroom), ^ 

(4) Action In rdation to verbal or nonverbal sounds (e.g„ giving a briefing, 
playing a bi^le). 

» { 

These means of leamingH;he student activities of observing, reading, listening, and 
doing-are used in the following section for selecting sqjproeriate training methods. 

DEFINITION AND CLASSIFICATION OF TRAINING METHODS 

Most research results that were intended to indicate the relative effectiveness of 
instructional methods have been confounded because the researchers fitiled to isolate the 
elements of methods that governed learning activities. In other words, tiSe methods used 
were not carefully defmed and controlled so that specific results could be attributed to 
the effects of qf>ecific elements or to different interaction effects of the elements. To 
avoid these errors in future research, it is necessary to carefully define and classify the 
dements of methods according to their basic function. 

DEFINITIONS OF METHODS AND TECHNIQUES BY FUNCTIONS , 

The defmitions in this section are classified according to three major categories: 

(1) Pure presentation methods and tediniques t 

(2) Student verbal interaction methods and techniques 

(3) Knowledge ^plication methods 

The term 'liedinique*^ implies to those instructional functions that cannot stand alone as 
methods, but when used m conjunction with some methods may increase method 
effectiveness. These techniques are identified as sudi in their definitions. 

The names given to the methods and' techniques are not necessarily intended for 
Army use. For example, "lecture'^ is a term excluded from the lexicon of Army training, 
,90 it would not be used in mettiods-media selection materials for use by Army trainers. 
For the purposes of further researcbt ^iectare** is merdy a label for the function that is 
defined. 



Pure Presentation Methods and Techniques 

Lecture^ A formal or semiformal presentation of oral information by a single 
individual; facts, concepts, problems, relationships, rules or principles presented orally 
either directly (as by a classroom instructor) or indirectly (as by tape recorder, film, or 
TV); basically a means of telling students information they need to know to attain a 
training objective. 

Jemonstra Uon. Portrayal of a sequence of events to show a procedure, technique, 
6r operation; inay be oral (as in teaching radio procedures) or visual (as in teaching 
operation or handling of equipment or piaterials); may be presented directly (as by a 
classroom instructor) or indirectly (as by fihn or TV if visual, or by tape recorder if 
oral). 

Exhibit. A visual display used to present information essential or helpful to the 
student in attaining a training objective; may be, for example, actual equipment, models, 
mockups, graphic materials, displays (by Venetian blind or chalkboard), projected images, 
or sand table; hiay be presented directiy in the cla^pom or indirectly by film or TV. 

Indirect Discourse. Verbal interaction among two or more individuals which is heard 
by the student; may be a dramatization, such as a play or a skit, or a dialogue between 
pwel memb^foUowing flieir lecture presentations. 

Assigned Reading. Printed verbal materials such as books, periodicals, manuals, or 
handouts from which the student is required to obtain information essential or helpful to 
him in attaining training objectives. 

Rhetorical Questioning. ,A presenter technique of ^ usinjg questions, not to evoke 
student verbal responses but to emphasize a point, stimulate student thinking, keep 
students alert, or direct student attention, with the presenter provkiing his own answers; 
may be used directly (as by a classroom instructor) or indirectly (as by film or TV). 



' Student Verbal tntdraction Methods 

Evocative Questioning . A presenter-controlled technk^ue in which questions are used 
to evoke objective-related infonnation firom the student to akl his learning and to provide 
feedback to the presenter as to the d^j^ and quality of student learning. 

Programed Questioning. A presenter-controlled technique used to systematically 
demand a sequoice of appropriate student re^nses; nuty be used directly (as by an 
instructor in a classroom) or indirectly (as by film, TV, programed booklets dr teaching 
madiines, including computers). 

Student Query . A technkiue by which studente can search for information by 
questioning a classroom instructor, tutor, coach, or an appropriately programed 
computer. 

Discussion. Interactive sharing of information and experiences by a group of 
students; includes statements of ideas and comments related to achieving a Gaining 
objective; an insbructor may or ihay npt be present to monitor the exchange. 



Knowledge Application Methods ^ 

Practical Exercises. Student interactions W5tli things, data, or persons, as is necessary 
to attain txaining objectives; all forms of simulation (e«., games and interaction with 
hardware simulators) and interaction with actual equipment or job materials (e.g., forms). 



The foregoing statements, proposed as mutually exclusive definitions of training 
methods and techniques, are sufficient to include all training processes. This does not 
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mean that the tenns used must be substituted for other terms that have traditionally 
denoted ^'methods'* of training or instruction. It is helpful to note, however, that 
traditional terms can now be mote clearly defined by stating the mutually exclusive 
functions, as defined earlier, which are included in traditional methods. 

A conference^ as the term* is understood in Army training, may be defined as a 
training process that includes lecture, exhibits, evocative questioning, rhetorical ques* 
tioning, and student query (or whatever function that may be included). 

€uest Speaker (used by the Army to denote a method of instruction) can be 
defined as a lecture and whatev^ other function or functions are used dur'Jig the 
presentation. - 

A seminar (depending upon the specific processes eadi seminar may include) 
may be defined as a discussion following instructor or peer lecture or demonstration. 

A case study may be defined as a discussion foUowing a presentation by 
reading assignment, lecture, or demonstration. 

Role playing may be defined as a practical exercise and may include indirect 
discourse, evocative questioning, and discussion^ 

iVoiptimed instruction may be defined as including programed questioning and 
exhibits (as In Skinner and or, as in ttie case of adjunctive 

prognmtied instruction^ assifpied r^ 

TYitorin; or coaching may be defined (depending upon the function of the 
tutor or coach) as questioning, lecture, demonstration, and student query. 

Pter mstruction—instruction by a fellow stUdent or students— is not sufficiently 
defined unless the peer^instruction function is designated, sudi as, demonstrating, 
questioning. 

Correspondence instruction may be defined as assigned reading and posubly 
practical exercises and, perhaps, programed questioning. 

This list of terms which traditionally des^pfiate training methods, but which actually 
denote one or more of the mutually exclusive functions as defined above, is not intended 
to be exhaustive. ^ ^ 

RELATING MEANS OF LEARNING TO TRAINING METHODS 

Teaching points can be classified according to their means of teaming, which are 
observing, reading,' listening, and doinij. To use the means of learning to select training 
methods, a device is needed for identifying the methods^y which eadi means of learning 
can be implemented. Table 3 is sudi a device. ^ 

In Table 3, two methodsHlemonstration and exhibit-are subdivkled to distinguish 
between two kinds of demonstrations (visual/oral) and two kinds of exhibits 
(visual/print). Also, it can be seen that ttie learning requirements of observing, listening, 
reading, and dohig can^ be implemented by various methods, ranging firom one fordoing 
(but of many4ypes) to four for listming.^ , 

The techmques of rhetorical questioning, evocative questioning, programed ques- 
tioning, and student query do not sppevt in the table since they cannot stand alone as 
methods, and because the means of learning are not sufficient criteria for their selection. 
(Criteria that may be 4>propriate are discussed in the following section. 

INDIVIDUAL DIFFERENCES AS CRITERIA FOR 
METHODS/TECHNIQUE SELECTION 

Selection of the niost satisfactory instructional methods and tediniques may 
ultimately depend upon the requirements of student indivklual differences. 
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Exhibit (Visual) 
Exhibit (Print) 


X 
X 


X 


X 




Indirtct Discourse 
Assighad Reading 
Discussion 
Practical Exercises 
(Many Types) 




. X 
X 


X 


X 



Certainly, every person is different, and some authorities, such aTCroiibach and 
Snow (4) and Bradit(&), have suggested that no single instructional process will provide 
the most effecthre learning experience for all studmts. They note that some students wUl 
be more successful with one instructional program, others, with another, in achieving a 
common set of objectives. But research dealing with individual differences has faUed to 
yield sufficient data to provide an empirical basU for selecting the most effective 
methodsHnedia combinations. For example, G^cht reviewed 90 research studies and 
found only five that showed methods to be differentially effectwe with different groups 
of students, and these differences were all associated with verbal interactive methods. 

Hoban and Van Ormer (6), in a summary of film research, found some evidence to 
suggest that learning from films is dependent upon student intelligence, training, ot 
pttvious knowledge of the subject. They further suggested that higher ability studente 
gain more from verbal presentations. Meierhenry (7) and Smith (8), however, asserted that 
evidence for the superiority of fUms for low IQ"studente was not warranted. But more 
recenUy, Snow and Salomon (9) concluded that film and television presenUUon charac- 
teristics are likely to interact differenUy with studente of varying aptitudes. 

. Numerous researchers, such as AUen and D*ehling(10), Eattu(n^and Baker (12), 
have been unsuccessful in using the Guilford (13) structure-of-inteUect abUity factorslo 
find differences in abUity that predict the effecUvenew of different methods and tech- 
niques with students who differ in abUity. The resulte of these carefully controlled 
ocperimente are well summarized by Bunderson and Dunham (14), who expressed doubt 
about the practical values of using different abilities to differentiate between methods. 
They give four reasons for their skepticism: 

(1) Individual difference utcful in predicUnf differencec in method are very 
uncommon. 

(2) The differencec which are found are not large enoi«h to be useful, becauie 
slii^t dtanges in th^ learning taska eliminate the differences. 

(3) The criterion-based tesU used to evaluate performance do not enable them to 
be used in diagnosing possible differences. 

(4) The payoff gained by providing alternate roethodt may be leu than the 
payoff gained by revising the single best learning task. In instructional design 



Ihm it probably no factor that produces more improvtmcnt for mort " 
studtnts than the cartful revition of a program bated on an analytit of 
ttudtnt ptrformanctt. 

Other reteardittft have not been as pesnmistic at Bundertpn and Dunham. »For 
example, Allen (15), after reviewing the literature in tMching methods and media, 
concluded that study of the three*way interaction between stimulus, task, and the 
indhridual learner, while extremely complex, may lead to a more precise understanding of 
the initructional roles of methodt and media. The potmtial fruitfuhiess of sudi researdi 
hat been suggetted by Gagne (2), Briggt tt oL (16), Salomon and Snow (17), Briggt (18), 
Allen (15), Salomon (19), and Campeau (20). Alttioui^ no adequate empirical evidence 
hat been generated to date, there it reason to attume that fiirther research diould provide 
the criteria necettary for tdec^ the:moit satisfactory methodt and techniques for ghren 
purposes. : / 

In HumRRO*s Pkoject IMPACT, for example, ttie significant differentid performance 
imdictors in tiie introductory part of tfie course were different firom those ttiat were 
useftd for prediction of performance in the later, more complex stages (Seidd, 21). In the 
introductory part of the course, five ftetors proved to be signifidmt predictors; these 
induded associative oir primitive memcvy, gmrat reasoning, a gatienl quantitative skill, 
and student npectancy just prior to the oiterion test In the li>i^« men complex 
portion of tfie coum, 11 fMtors were 'rfiovm. Figural id^[>tive fl^bility (the main 
factor), peroep^ud ^eed, and aUlity to make comparisons rapidly and accurately 
appwred especially useftd* Student expectancy (sdf*assessment) appeared important at an 
forUer point in this porti<m at the pre-test. Finally, a higher levd memory factor, * 
chunking memory,, was showm 
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Chopttr 3 
MEDIA 



MEDIA SELECTION AND MEDIA DESIGN 

IMia idtction hat two nlatad, but veiy diCfeitnt» aspects: (a) madia selection per 
se, whidi is dioosiiif betwem the physical (material structure) diaracteriitsc« devices 
fliat can be used to convey mestsgsi; and (b) media design, whkh is sdecting the 
particular attributes of media (such as oi«)abiUties of presenting color or of depicting 
motion) that can most effectively present differsnt kinds of teadiing points and assist 
student letfning. 

Madia derign to fsr mqn important than nmdy sdecting the phydcal medto that 
have or can be made to ham osrtain attributesy In other words, as Tosti and Ball (1) 
suggested, selection of media equipment and devices should depend upon the attribute 
that are neeeisary fdr effective in$truction. 

* * 

INSTRUCTIONAL MEDIA REQUIREMENTS 

Some instructional medium or combination of instructional media (irtiether persons, 
hardware, or softvnue) to essenttol fdr presenting knowledge that the student must leant 
In addition, instructional media must gukle and otherwise asstot ttie student in learning, if 
learning is to be efficient » 

Gagne*s (2) anelysto of learning and media ftinctions greatly akto understanding of 
^Micific^medto requtoements* He tentativdy klentified eii^t types of learning and pointed 
out that the basic ftanction of initeuctional medto to to cusatt (At external learning 
eonditlone that art necessary for eadi type of learning to occur efficiently. More 
ipedfically, ht noted that to create the external conditions required for learning, medto 
must do seven things: 

f (1) ftesent stimuli. Ever, tht simplest type of Iteming requires a stimulus, and 
as the types of learning become more cmnpltx, various kinds of stimuli art necessary to 
make learning both posribto and tificitnt 

(2) Dtofet studwit activitfts . Various kinds of stimuli must direct student 
attention and ottierwtoe indicate to tht student vriuit he must do in the learning situatim. ' 

(3) Ptt>ykle raodds fbr terminal performance. Such modds orient the student 
to the nature of the behavior he must learn. 

(4) Proigde prompts. For most kinds of lesiming, prompts should be available 
to the student. For exampltl 'earning of verbal and motor diains to made easier if 
demonstrations or modeto provide prompts to help learning of proper sequences. Then, 
too, midti|rie discrimination learning to made easier if prompts or cues make differences 
in woids or things more dtotinctivt. 

(6) Guidt thinldng. Efficient learning of mental skiUs requires hints or sugges- 
tions to guide student thinEing.^8udi guidance hdps tht studtnt to makt connections 
bttwttn conctpts whidi ht can ust to form princi|des, and to discern relationdiips 
between prindples whidi he can use to solve problems. 
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(6) Induce transfer of knowledge. The process of learning to transfer knowl- 
edge to novel situations usually is begun by asking the student questions of the 
problem-solving variety. Demonstrations may also aid this kind of generalizing (transfer) 
to other things or situations that are similar to« but not the same as, those used to teach 
the concepts or principles or those used to demonstrate how the knowledge can be 
generalized. Visual exhibits also can induce transfer by illustrating the kinds of things or 
situations to which the knowledge applies. ^_ 

(7) Piovide feedback. An important condition for learning is to let the student 
know whether the consequences of his learning are correct so that the learning can be 
reinforced if correct or be restructured if incorrect. 

Sudi instructional media requirements indicate how media can be ustu > create the 
external learning conditions which -make learning possible and efOcient. Obviously » media 
am be used in many ways, but to reliably provide the most satisfactory media for given 
purposes, one must have adequate evidence of the effectiveness of various media 
attributes for different purposes. 

GUIDELINES IN THE LITERATURE FOR MEDIA SELECTION 

Research results and^theoretical concludons by instructional-media authorities con- 
cerning the form and manner of {nreSenting teadiing points provide some '(though 
incomplete) guidelines for media selection. The various considerations are discussed under 
the following groupings: 

1. Words (reading/listening) or pictures 

2. Words (reading/Iistenii^) and pictures 

3. Pictures 

4. Listening or reading 

5. Listening and reading 
^.^ Listening 



WORDS (READING/LISTENING) OR PICTURES 

•The nature of the knowledge (teaching point) to be learned will, in large part, 
determine tiie form and manner of presentation, such as by telling or by showing. 

A decision to use words or pictures usually is based upon whether the knowledge 
can be presented concretely (Taylor, 3). Allen et d. (4) demoiistrated the ease of making 
this distinction v*hen they found that subject matter coukl be identified by untrained 
persons as inherently concrete or non-crete. Allen et al. {S) report, however, that little 
specific direct study has been made of the effects of using words or pictures, although 
mudi attention has been given to different types of media that have words or pictures as 
elements. 

Given the pr^nt state of knowlec^e in this area, Gagne and Rohwer (6) concluded 

that: 

(1) When given the diotce between words and pictures (if deemed equivalent) 
pictorial materials are superior to verbal materials. 

(2) The conditions that mi^t dictate choices between ushng words and pictures 
are almost entirely undermined at the empirical level. (They were not aware, 
however, of the ^len et at, (4) study.) 
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Levie and Dickie (7) have sununarized wme theoretically generated guidelines for 
selecting words or pictures: 

> Pictures usually are superior to words for eHctting recall and recofnition. 

^ ' .(2) Pictures usually are superior to words iHien tbe student lad» the verbal 

equivalent (when the student knows no word or words to reptesent a thing - 
oreonceptX 

(3) Pictures usually are siqjierior to*words for teadiinf concrete subject matter. ^ 

(4) When learning involves understanding a large number of r^htiondiips, a visual 
exhibit which presents all the reiatiomhips simultaneo^^^y in a structure 
(sudi as a nM^ or schematic diagram) <£an be helpful. 

(5) Learning of behaviors sudi as creative thinking may be assisted by visual 
di^lays. 

(6) When the order of ideas or knowledges is critical, words are preferred. 

(7) When graphic materials are not readily interpreted by the student, spoken 
_ words are preferred. (If graphics are to be usefol, the student mtlst be sldUed 

^ in interpreting them.) ' ^ ^ 

One must keep in mind that ttiese guklelines were theoretically, not empirically, 
derived* 



WORDS (READING/LISTENING) AND PICTURES 

Levie and Dickie (7) concluded ttiat more learning mi^ result from audiovisual 
presentation under certain conditions, but tiiey were unable to define or understand these 
conditions, usually because the resewrdi from ndiidi they drew ttieir conclusion was not 
well designed and controlled. 



PICTURES 

To determine mhm pictorial or graphic materials should be used to present teaching 
points, one must consider irriietiier the following media attributes are necessary: color, 
motion, realistic detail, multi-image, and size of illustration* Studies related to each of 
ttiese variables are summarized in ttie following paragraj^s* 

Color, Exton (8) noted that color is used to distingui^ between or to identify 
dements in graphic displa]^, to hi|^ilight significant features, to serve as a basis for codes, 
and to darify ttie entire presentation by providing visual contrast. The effectiveness of 
using color to differentiate between items in diq^lays to reduce search time has also been 
documented by many researchers, most recenUy Shontz et aL (9)* ^ " 

Use of color as potential luds in learning is indicated in study reports by 
Peterson and Peterson (10), Saltz (11), and Weiss and Margolius (12). One must be 
careful, however, to use color iq[ipropriately in training* For example. Underwood (13), 
Mechank: (14), and Bahrick (IS)-found that performance suffered when color was used in 
training but not in tiie task to^ yNhkh the tiraining was to transfer, because the students 
learned and responded to color distinctions (cues) in (raining that were not in the task 
situation* 

Motion . An excellent study of tiie relations of motion to learning was made by 
Allen and Weintraub (16)* They examined learning of facts, sequences^ and concepts, and 
found that motion was best used for learning procedural sequences. They also found that 
motion portrayal akled learning only when inotion was an integral part of what had to be 
learned* The implications of ttieir study are that motion shouU be used when (a) the 
content to be learned consists of tiie motion itself; (b) diaracteristics of th movement 
must be learned; and (c) the content is enhanced and clarified by tiie motion. 
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Spangenbei^ (17) examined the effects of motion in learning a weapon- 
disassembly procedure. He concluded that motion helps learning of procedures when 
(a) the activity requires simultaneous motion in different directions; (b) the activity^ is 
unfamiliar to the student; and (c) the activity is not easily described in words. 

Degree of Realistic Detail . Dwyer (18) summarized an extensive series of studies on 
degree of realistic detail in illustrations. He found in pace studies (when all students 
received equaTvijswing time) that illustrations showing relatively small amounts of realistic 
detail were mo'st effective. However, in self-paced studies (as much time as the student 
wanted), .he found that the more realistic illustrations were most effective. Dwyer 
concluded, therefore,* that effectiveness of the d^ee of detail in pictorials depended 
upon whether study time was limited by the instructor or unlimited according to the 
student's choice. 

Dwyer also concluded that effectiveness of an illustration in helping the student 
depends upon, the type of information he needs; for example, if he must learn to sketch 
an object, he should have a line drawing instead of a photograph. In other words, as 
Denenberg (19) also showed, the illustration should provide only that degree of realistic 
detail that is required for transfer of training to job performance; unnecessary detail 
reduces the efficiency of learning. 

In studies of training men to perform procedural tasks. Cox et al. (20) and 
Grimsley (21,22,23) found that fidelity (realistic detail) of training devices can be low 
with no adverse effect on training time, level of proficiency, amount remembered oVer 
time, or time to retrain. 

Cox et al. also found no differences in learning a procedural task on a full-sized 
panel, a half-sized panel, or a panel only one-nineteenth the area (3x7 inches) of the 
full-sized panel, if the parts of the device remained clearly visible. 

Size of Illustration. Moore and Sasse (24), in a study comparing the size effects of 
projected illustrations, found that recall of detail was best when a medium-sized screen 
area of 35 x 70 inches was' used. (The large-sized area was 70^x 70 inches, and the small 
^ size was 35 x 35 inches.) 

Other studies have not shown that reliable differences in effect are attributable 
to the size of screen area (Greenhill, et al 25; Reede and Reede,26). The illustration 
must be large enough, however, to show the detail required for learning. 

Multi-Image Displays. Millard (27) suggests that for instructional purposes the 
projected. multi-image is particularly adapted for making comparisons, for illustrating the 
devdopment of, related concepts, and for showing relationships. Although logical, his 
suggestions are not based upon empirical ^e^vidence. Lawson (28) reported that multi-image 
displays affect attitude and meaning. He based his conclusions on his observation of 
student reactions. 

LISTENING OR READING 

Sticht (29) foun(l that a large number of men of poorer reading ability said they 
preferred to learn by listening rather than by reading. He also found indications that poor 
readers might learn certain prose materials as well by listenings as by reading, and he 
concluded that listening as a skill has special meaning for inept readers. In a later Army 
study, Sticht et al (30) found no difference in learningi?by listening and by reading. 

After reviewing the research, Hartman (31) concluded that reading is more effective 
than listening when the information is difficult or complex. He also found several specific 
instances with less difficult material when listening was best. 
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LISTENING AND READING 



Simultaneously reading and listening to the same words (total redundancy) does not 
facilitate learning of V^i^al materials-^probably, as' Carroll (32) suggests, because oral 
presentation tends to ^be much slower than normal reading speeds, and the reading and 
listening interfere witi^v.|a^ other* 

I Allen et at. (4) fouhd that directive audio in conjunction with printed materials can 
aid learning if it Is used to discu^ and emphasize parts of a message or of a printed 
exhibit. 



LISTENING ^ 

Chu md Schramm (33) cite several studies that compare indirect discourse with 
lecture. They concluded that there are no reliable differences between these two 
methods* : 

Stidit et a/, (30) found that with Army students as nauch as 36% savings in listening'' 
time can be attained with only, moderate loss in comprehension by using rate-controlled 
recordings (compressed speech).^ 

Carroll (32) cites studies of many other variables involved in learning by listening 
that should be considered in the design of listening experiences, but that are not useful in 
differentiating between either methods or media. 

Although some of the data mentioned in the Guidelines section can foe xiseful in the 
selection of media, further research. is needed on the effectiveness of media attributes for 
specific instructional purposes before an empirical basis for selection of the most satis- 
factory media can be derived. 



DEFINITION AND CLASSIFICATION FOR ADEQUATE MEDIA SELECTION 

Many researchers have expressed the need for clear definitions of instructional media 
to aid unambiguous communication. Such definitions are also essential for development 
of an adequate media selection procedure. In addition to unambiguous definitions, a 
viable media classification system is ui^ently needed, as a basis for further fruitful 
research and for a feasible media selection procedure. Neither of these needs is met in the 
literature. 



MEDIA DEFINITIONS 

Several lists of media definitions are in the literature, such as those provided by 
Gerlach and Ely (34), E^own et al (35), and Bretz (36), but all have the following faults: 
(a) Only a few of the same media and media-related terms are defined in each list; (b) the 
terms common to all lists are defined differently in each list; (c) no clear criteria are 
given for listing or excluding terms. 

To be adequate for selection of media for specific instructional purposes, the list of 
media definitions must be prepared according to the following rules: 

(1) The list of terms must be exhaustive. 

(2) Media deHnitions must be based upon meaningful media attributes 
(capabilities). 



(3) The definitions must be easily understood. 

(4) The definitions must relate directly to user-identified media requirements. 



MEDIA CLASSIFICATION 

For a madia classification system to be viable, it must satisfy the following 
requirements: 

(1) The number of categories must be large enough to include all media, but 
small enough to make the selection procedure easy to use. 

(2) The categories must be defined^Jn^Jerms that clearly relate to user 
requirements, and thus: 

(a) Appropriately order user thinking about media 

(b) Provide the means for unambiguous communication between systems 
engineers and others involved in designing and developing training 

programs.*^ * 

^ - -* 

In other words, adequate media classification must, as Meredith (37) stated, provide 
utility, convenience, and economy. 

Although numerous attempts have been made to classify media (Appendix B), none 
has met the preceding requirements, and none can be adequately adapted for Army use. 

A PROPOSED SCHEME FOR MEDIA DEFINITION, CLASSIFICATION, 

AND SELECTION 

THE AGGREGATIVE DEFINITION 

The lack of useful media definition and the analysis of definition requirements 
sparked the notion of the "aggregative definition.'* This term denotes that, for practical 
purposes, an. instructional medium is nothing more nor less than the sum (or aggregate) 
of its functional capabilities in a training process. Thus, aggregative media definitions are 
precisely what is needed for use in selecting media. 

To provide sudi definitions, note first that instructional media have two basic 
functions: (a) to implement training methods, and (b)to assist student learning of 
teaching points. 

Therefore, to state a complete aggregative definition of any medium, one must 
identify its capabilities for presenting: 

(1) Stimuli that can ])e observed, read, heard, touched/felt, tasted, or smelled 
(to implement training methods). 

(2) Stimuli that can assist the learning processes of association, discrimination, 
and generalization. 

(3) Stimuli that can assist students of different characteristics in learning 
different kinds of teaching points. 

Identification of media capabilities that can implement training methods is a rather 
simple process that Army systems engineers and instructors can perform. But 
klentification of media capabilities that can best assist learning has not been done with a 
satisfactory degree of confidence by even highly skilled instructional media specialists. 
One reason is that, as Saettler (38) pointed out, not enough experimental evidence exists 
to reliably show the different effects of various media stimuli on the learning of different 
kinds of teaching points by different kinds of students. By "media stimuli** we mean 
such things as color, shaded contrasts, symbols, graphs, regular motion, speeded motion, 
slow motion, photographs, montages, depictions of time and distance jjtelationships, 
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enlarged or reduced sizesi animation, tachistoscopic flashes, X-rays, and two and three 
dimensions* 

This is not to say tiiat nothing is known of the effects of media, but rather that the 
state of this knowledge is general rather than specific. As Salomon (39) noted, it is 
known that the effects of media attributes vary when the subject matter varies, and when 
the soidents vary in abilities, attitudes, and cultural background. In other words, only 
enouf^ is known for us to realize that there are problems whidi further research must 
solve if enough specific media effects are to be klentified for reliable selection of the 
most satisfiu:tory media. 

MEDIA CLASSIFICATION 

To pro'/kle a basis for a media selection procedure, media attributes must be 
classified in a way that creates a media selection qrstem* The media attributes useful 
folr our purposes fUl into three major categorier-media which can be used to: 

(1) Implement training methods. 

(2) Assist the learning processes of association, discrimination, and 
generalization. 

(3) Assist different kinds of students, in learning. 

These statements are easily translated into three major selection categories by which 
the media attributes must be classified: 

(1) Training methods 

(2) Learning processes 

(3) Student differences 

Eadi of these categories must, of course, be divided into subcategories so that media 
attributes can be classified according to specific methods, learning processes, and student, 
differences.' 

' Tables 4, 5, and 6 sKow how media attributes (aggregative media definitions) can be 
used to relate media to the mi^jor selection categories. For illustrative purposes, eadi 
category is represented by only a few examples. The complete system would include all 
useful categories and subcategories. 



SELECTION PROCEDURE 

Having created a classification system that can be used for sdecting media, the 
selection procedure can be derived by merely stating how the system woukl be used. 

. Although methods and media have been discussed separately and a methods«selection 
procedure apart from media has been proposed, it should be noted again that for all 
application purposes neither methods nor media can be consklered separately. As 
mentioned in Chapter 1, methods have no reality until they are jspplied by the media 
that implement them. In other words, to ask ''Which method is most effective?*' and 
''Which media are most effective?'' is to pose pseudo questions, for they cannot be 
answered separately. The only meaningful question is, "Which methods^nedia combi- 
nations are most effective for givm instructional purposes?" 

'or interest are the findings of HumRRO Army studies in Work Unit SPECTRUM on the effec* 
Uvenett of different traininf ttratef iet in aiding different learning proceMet with men of different apUtude 
levels. They are reported by Showel (40), Voxei «/. (41 ), Montague and Showel (42), Taylor and Pox (43), 
Taylor and Montague (44), and McFann (46), 
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Table 4 



Media Classification According to 
Capabilities for Implementing Methods 





Media Stimuli* 




To Be 


To Be 


To St 


Tr«imn9M«thod 


Obitrved 


Haard 




Lecture 




X 




Demonstration (visual) 


X 






Demonstration (oral) 




X 




Visual exhibit 


X 






Print exhibit 






X 


Indirect discourse 




X 




Assigned readirig 






X 



*Each "X" rtpritents the namM of all madia that have the capabilities 
for implementing the training mtthodt. 



Table 5 



Madia Classification According to 
Capabilities for Assisting Learning Processes 



turning ProcMS 


MtdiaStintuli* 


Color 


1 Motion 1 


Sound 


Association 


X 


X 


X 


Discrimination 


X 


X 


X 


Centralization 


X 


X 


X 



*Each "X" rapretantt the naniM of all madia that have 
the capabilities of prtienting stimuli to assist the learning 
proctssts. The catagories of color, motion, and sound are 
only exampttf . Further research must identify at! stimuli 
that should be ratatad to the learning processes. Alio, each 
major laarningiKOctit category must be dividad into 
subcategories for the kinds of relating, discriminating, 
and ganeralizing indicatad by various kinds of teachitig 
points. 



Therefoie, the mettiods-selection proceduie proposed in Chapter 2 and the 
media-selection procedure proposed here must be combined as shown in T^ble 7. 

The medium or media identified (together with their attributes) may con^ of 
hardware, or software, or both. If hardware is involved, it may be on hand in the school 
that will use it; if not on hand, a decision of whether to buy it can be made. When 
software is invoked (ejg., film, videotape, print), the training materials to be mediated by 
the software may be available; if not, a decision of whether to develop it can be made. 
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Tables 



Media Classification According to Captbilities for 
Assisting Different Kinds of Students in Learning 



Studtflt Characttrittic 


Mwtia stimuli* 


Color 


1 Motion 1 


Sound 


Agt 


. X 


X 


X 


Ltvel of Education 


X 


X 


X 


Ltaming Ability 


X 


X 


X 


Other 


X 


X 


X 



'Each ''X" rtpmtnts tht namti of mtdk. Tht maior 
citaQOfiM of ttudtnt difftrtncft art only axMnplM. Furthar . . 
rtMtrch miy axlffNl tht liit* raduce it, or aliminata it, dapand* 
in9 on twhathar cartain stimuli actually incrtata tha affactivanaw 
of inttniction with diffarant kinds of stMdahts. Also^ aach ttudijint* 
diffafanoa catagory must ba aubdividtd to'provida ^tagorias 
for diffarant aga groups* lavals of aducation, and laarning abilities. 



Thut» media selection consists of either identifying useful media that are available, 
or identifying media attributes that can be used to develop desired training materials. So» 
obviously, the selection procedure can be used for both media selection and media 
design. 

As noted eariier, Tosti and Ball (1) suggested that one should specify the media 
capabilities necessary for the insteuctional requirements before selecting the media. This^is * 
a difficult task to do well, for it requires L'udi knowledge of how 9)ecific media 
d^Mbilities can be used best. When future research provkles such knowledge, and when it 
is stored in the proposed media-selection system, the task of designing the most satis- 
factory media for given purposes will have been done. All the user would need to do 
woukl be to klentify the kind of teaching points to be learned, the kind of students who 
must learn them, and the method to be usM, and thai follow the proposed media* 
selection procedure^ 

By completing this relatively simple process, the user would be able to identify both 
the q[)propriate media capabilities and media devices. When two or more media have the 
same ciqpabilities (which probably woukl often be the case), an additional selection step 
woukl be necessaryHhat of dioosing between the media on the basis of cost- 
effectiveness* This dioice, however, as well as decisions to buy or to develop media* 
should be made only after cpst^ffectiveness studies and utility analyses (discussed in 
Fart 11) have been made. 

THE AGGREGATIVE DEFINITION CARD 

-t 

The numbers of media and media attributes that must be classified make the use of 
matrices (similar to classification Tables 4, 5, and 6) impractical, as the selection 
procedure woukl be too complicated and time-consuming. 

If, however, edge-pundied or IBM cardii were used, the selection procedure would be 
(piick and easy. Edge^undied cards hatve a row of holes along each border. The 
classification system would be established by assigning a specific card hole to each of the 
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Table? 

CombiMd Mtthodt-Mtdia Sttoction Prqotdurt 



Stitction Prootdurt 



Product 



Methods 



1. Apply rules for relating teaching* 
point characteristics to nneans by ^ich 
the teaching points can be learned. 

2* Use the particular nwans of 
learning to identify nwthods that . 
can implement that means. 

3. Use suident difference charac- 
teristics to identify method most 
effective for a specific kind of 
student 



Media 



1 . Uie most effective method 
identified above to identify media 
that can Implement that method* 

2. Use the same teachingixrint 
characteristics used to identify 

a particular means of learning to 
more specifically identify appro* 
priatem«dia. 

3. Use the same student* 
difference characteristics used 

to identify most effective method 
to also identify the most effective 
media. 



A particular means of learning 
specific teaching points (observ* 
ing, reading, or listening). 

All training methods that 
can implement the means of 
learning. 

The most effective method 
for presenting the teaching 
points to specific students. 



All media thatican implement 
the method and assist learning. 

All media that can implement 
the method and assist learning 
of the specific teaching points. 



The medium or media that 
can be used for both imple- 
nwnting the method and 
assisting a particular kind 
of student in learning the 
specific teaching points. 



subcategories of the niajor categories we have identified as methodSi learning processes, 
and student differences. Eadi medium wouU be represented by one card; if there are 66 
media, for example, there would be a stack of 65 cards* 

On the face of each card would be the following (and perhi^ other) useful 
information concerning the medium: 

(1) Generic name 

(2) Attributes 

(3) Special connderations for use 

(4) Space and facility requirements 

(5) Personnel requirements 

(6) Cost factors 

il) Effectiveness factors 
A medium card would be coded for selection purposes by the following process: 
When a medium attribute is retoted to one or more of the method s leaming«process. and 
studentdifference subcategories, the hole asdgned to eadi of these subcategories would 



be notched out on the card. When all appropriate holes are notdied out, Jhe medium 
card will have beeiL completely coded for sdection. f 

To select the most effective media for given instructional purposes, one would use a 
card-sorting needle and perform the following three steps: 

(1) To select all media that can' implement a gwen method, the user would 
insert the needle in the card hole assigned to that mettiod. He would then raise the 
needle, and all appropriate media cards would fall out of Jhe deck. He would use only 
these cards for the next step. 

(2) To select those media that can both implment the method and assist a 
particular learning process, the user would insert the needle in the hole assigned to that 
learning piceess, raise the needle, and the i4)propriate cards would foil out for use in the 
final step. 

(3) To select those media ttiat can implement the method, assist the learning 
Fit>ces8, and assist a particular kind of student in learning the teaching points, the user 
would insert the needle in the hole assigned to that particular kind of student, raise the 
needle, and the appropriate card or cards wouU fall out 

It only one card is obtained in the final step, the most satisfoctory method-medium 
combination has hem klentified. If more than one card falls out, the. most satisfactory 
metiiod*medium combination coukl be selected on the basis of cost-effectiveness and 
utility analysis data. 

NUMERICAL CLASSIFICATION METHODS 

Several numerical methods for classifying large numbers of variables have been 
developed in various areas of science, eq[>ecially in biology (Sokal and Sneath, 46), and 
these methods have proved to be powerful dassificatioh tools. 

Silvermsin (47) ^^^amined the fmibility of sudi tediniques for task classification and 
concluded that diSr use is wananted in solving {nrbblms related to Navy personnel 

McQuitty (48, 40, 50, 51, ^,53) devised several numerical methods for classifying 
experimental subjects (persons) accorclmg to large numbers of experimental variables. 

One or inore of these methods may well be adapted for classifying and selecting 
media if further research proves the proposed approach to be inadequate. 



References for Chapter 3 

I. Toiti, D.T. and Ball, J.R. "A Behavioral Approach to Inatmction Design and Media Selection/* 
AudiO'Viiual Communication Review^ voL 17, no« 1, 1969» pp. 5-25. 

2* ' Oagne, KM. The Conditions of Learning, Holt, Rinehart and Winiton, Inc., New York. 1965. 

3* Taylor. I.A. "Perception and Design/' in Research Principlei, and Practices in Visual Communication, 
J. Ball and F.C. Byrnes (ed$.), NBA, DAVI, Washington, D.C., 1960, pp. 107-118. 

4. Allen» W.H. et ah Visual and Audio Presentation in Machine Programed instruction, OE Final Report, 

Researdi Division, Department of Cinema, University of Southern California* Los Angeles. 
January 1967. 

5. AIIen» W.H* et aU Effectiveness of Different Combinations of Visual and Verbal Presentation Modes 

in Teaching Different Kinds of Learning Tasks, OE Final Report, Research Division, Department 
of Cinema, University of Southern California, Los Angeles, September 1970. 

6^ Gagne, R.M. and R<^ wer, W.D. "Instructional Psychology Annual Review of Psychology, vol 20, 
1969, pp« 381^118. 

7. Levie. W.H* and Dickie RE. 'Tlie Analysis and Application of Media,** in Second Handbook of 
Research of Teaching, RM.Vi. Travers (ed.), Rand McNally, Chicago, 1972. 



29 



8. Exlon» W.R. Audhviiual A9d$ to !n$truction, McGiiiwHilt Book Co., New Yoric, 1947. 

9. ShonU, W.D. et aL "Color Coding for Infomaiion LoctUon/* Human Foeton, yol. 13, no. 3, 197 1 • 

pp. 237-246. 

10. PeUrson, L.R. and Ptteraon, M.J. ""Tht Role of Context Stimuli in Vtibal Lctming/' Journal of 

Experimental Psychology, vol. 53, no. 2, 1957» pp. 102405. 

11. SalU, B. "Compound Stimuli in VarM Learning: CogniUve and Stntory DiffemitiaUon Vmut 

Stimulus Selection/' Joumat ofBxperimenialPeychology, vol. July 1963. pp. 1-5. 

12. Weitt, W. and Marg oUua O. 'The Effect of Context Stimuli on Leaminf and Retention/' Journal 

of Experimental Peyehotogy, vol. 48, 1954, pp. 318-322. 

13. Underwood, B.M. "SUmulus SelecUon in VerM Leaminf in Verbal Behatfiorand Learning, C.N. 

Cofer and B.S. Muigrave (edt.), McGraw-HUI Book Co., New York, 1963, pp. 33-47. 

14. Medianic, A. "EfTectii of Orienting Ttek Practice and Incentive on SiniulteneouelncMentel and 

Intentional Learnlng/'Vouriial of Experimental Psydiohgy, vol. 64, 1962^ pp. 393-399. 

15. Bahricic, H.P. "Incidentel Learning Under TWo Incentive Conditiont,*VoiirfMlo/£xiMrtmenrel 

P^cftology, vol. 47, 1954, pp. 170-172. — 

16. Allen, W.H. and Weintraiib, R. The Motion Variable in FiUn Preeentathn, OB Final Report (Project 

No. S-1 123), Reaeardi Division, Department of Cinema, University of Soutliem California, 
Los Angeles, 1968. 

17. Spangenberg, R.W. ''The Motion Variable in Procedural Learning," Journal article In press. 

18. Dwyer, F Jil. ''Exploratory Studies in the Effectiveness of Visual Illustrations," A udio^Viiual 

^ Communication Review, vol. 18, 1970, pp. 235-249. 

19. Denenberg, V.H. The Training Effeetivenem of a Tank Hull Trainer, HumRRO Tedinical Report 3, 

February 1954. 

- 20. : Cox, John A., Wood, Robert O., Jr., Boren, Lynn M.» and Thorn, H. Walter. Functional and 

Appearance Fidelity of Training Devices for Fixed-Procedurei Taeki, HumRRO Technical Report 
65-4, June 1965. 

. 21. Grimsley, D.L. Acqui^tion, Retention and Retraining: Effects of High and Low Fidelity in Training 
Devices, HumRRO Technics! Report 69-1 , February 1969. 

22. Grimsley, D.L. Acquisition, Retention, and Retraining: Qroup Studies on Using Low Fidelity Training 

Devicei, HumRRO Technical Report 694, March 1969. 

23. Grimsley, D.L. Acquirition, Retention, and Retrtdning: Training Category IV Personnel with Low 

Fidelity Devices, HumRRO Tedinical Report 69-12, June 1969. 

24. Moore, DM. and Sasse, B.B. ''Effect of Size and Type of Still Projected PicturHs on Immediate Recall 

of Content," AiMfjo-VitiMil Communication Review, vol. 19, no. 4, 1971, pp. 437-450. 

25. Greenhlll, L.P., Rich, O.8., and Carpenter, C.R. The Educational Effectiveness, Acceptability, and 

Feasibility of the Eidt^hor Large-Screen nieiHsion Projector, Division of Academic Research 
and Sciences, Pennsylvania Stete Unlvenlty, Univerdty Parte, January 1962. 

26. Reede, A.H. and Reede, R.K. Televising Instruction in Elementary Et^nomics, Industrial Research 

Bulletin No. 5, The College of Business Administration, Pennsylvania Stete University, 
University Park, 1963. 

27. Millard, W.L. "VUual Teaching AkU: Production and Use," In The Encyclopedia of Photography, 

Greystone Press, New York, 1964. 

28. Lawson, B.R. "Motivating witii Multi-Image at the IJjS. Military Academy: The Medium for the 

70's - and Ite Public Relations Side Benefits," Audiovisuab Instruction, vol. 16, no. 5, 1971, 
pp. 54-59. 

29. .Stidit, Thomas G. Learning by Listening in Relation to Aptitude, Reading, and Rate-Controlled 

Speech: Additional Studies, HumRRO Tedinical Report 71-5, April 1971. 

30. Stidit, Thomas G., Caylor, John S., Kern, Richard P., and Pox, Lynn C. Determination of Literacy 

Skill Requirements in Four Military Occupational Specialties, HumRRO Technical Report 
71-23, November 1971. 



30 



31* Hartman, F*fL *'8iiiglt and Multiplt Chaantl Communieatioii: A lUvitw of Rw a r c h and a Propotd 
Mod«]/* Audh^Vimal Cpmrnunkathn Bevkm. vot 9, 1961, pp* 235*262* 

32* Carroll, Lemming from VerM Dkeoum in Eiua^thfml M^dk: A Bevkm of the Utiratuft^ 
Rtpori RB-71*61, Educational Ttstinc Smiet, Princtton, NJ* Ociobtr 1971. . 

33. * Chu» 0*C* and Sdiramm, W. LtomUig from tekvkhn: WIM the ReatarcA Seys, National Aatodation 

of Bdttcational BroadeasUrs, WaAlngton, D.a, 1966. 

34. Oarladi, V. and By, DJP* Te&diitm awl Jfadte, Prtntict* HaU, Inc., Bngltwood QifCi, Ntw Jtraty, 1971. 

35. Brown, J.W., etuLAV InUruction Mtdia und Methode, (3rd td.) McOraw HUl Book Co., Ntw York, 1969. 

36. Brtts, R. A nxonomy ofCommwUeationMedki^ Educational Ttdinolofy Publicationt, Engltwood Clifft, 

New JarMy, 1971. 

37. Maiaditti, P. 'Toward a TWxonomy of Educational lladia,*Mudio*VlMial Communkeihn Kevkw. 

^. 13, Wintar 1965,* pp. 374*384. 

36. 8atUltr,P. **Daaifnand 8tltctibnTactort,**itat^iir of Muealionoi jRta«MTJ^ ^. 36, no. 2, 1966, 
pp.115426. 

39. Salomon, O. ''Cultural Diffarancaa in Raadinf and Undaratanding Oaographical Mapa,** papar 

prttantad at Amarican Education^ Raaaarch Aaaodation Annual Matting, Chia^o, 1966. 

40. ^ Showtl, M. Dtvtlopmtfif of Two Autowmted Progrmm for Toachkig If iUtery Ju$tkt to Men of 

VarhkM Aptitude Ltutb, HumRRO Ttdinical Raport 66*6, Juat 1966. 

41. Fox, W.L., T^yk«r, tJL, and Caylor, 43. Aptitude Uvel end the Aequi^ion ofShOle end Knomledge$ 

in a Vorit^ ofMUUery ffeining Teth$, HumRRO Ttcbnlcal Raport 69-6, May 1969. 

42. Montfgut, E JC. and Showtl, M. A Revkw of Combet Support Truinini. HumRRO Tadmlcal Raport 

•1*19, Dtctrtbtr 1969. 

43. Tiyk>r, J.B., and Fox, W.U Differentkd Aj^^ehee to Training, HumRRO Proftarional Papar 4747, 

Novtmbtr 1967. 

44. Tiylor, JA. and Montafut, B.K. The InterrektionMp$ of Ability level, Inetruetionel 8y$tem, end 

ShiUAequkiOon. HumRRO Profttakmal Papar 29*70, Daetmbar 1970. 

45. McFann, H.H. Training Stretegke end JndiMuel Differencee. HumRRO Proftttkmal Papar 12*71, 

Junt 1971. 

46. Sokal, R«R« and Sntatb, H.A. Frineipke ofNumericel Taxonomy^ W.H. Fraaman, San Frandaco, 1963. 

47. Silmman, J. New T^chniquee in Ta$k Anely$i$, Raaaardi Mamorandum SRM-6842, VS. Naval 

Panonnaf Rattardi Activity, San Diago, California, Novtnd>tr 1967. 

46. McQuiUy, L.L. **Siagla and Multipla Hiararchical Claaalfication by Radpiocal Paira and Rank Ordar 
TVpaa,** Bdueetional end Peyehohgieel Meoiurement. vd. 26, no. 2, 1966, pp. 253*265. 

49. McQuiUy, L.L. "Similarity Analyaia by Reciprocal Paiit for Discrtta and Continuoua DaU,** Ediica* 

thnel and PeyehologicelMeoiwrenunt, vol. 26, 1966, pp. 825-631. 

50. McQuitty , LX. ''Typical Analyaia,** Bdueetional end Peychohgitel Meeewrement. vol. 21, no. 3, 

1961, pp. 677-696. 

51. McQuitty, L.L. "Hiararchical Syndroma Analyaia,** EdiicaMonaf and PeyAohgieal Meaiurement, 

voL 20, 1960, pp. 293*304. 

52. McQuitty, L.L. "ComprAanalva Hiararchical Analytla,** FHucathnel end Peyehological Mee$urement, 

vol. 20, no. 4, 1960, pp. 805-816. 

53. McQuitty, L.L. "Eltmentary LInkagt Analytia for laolating Orthogonal and Obllqua Typaa and 

Typal RalavaQclaa,**£diicaribfM/4indPiyctok)f tol JtfaaMiJvman vol. 17, 1957, pp. 207*229. 



31 



r 



is" 



Port II 

ANALYSES OF TRAINING COST-EFFECTIVENESS FACTORS 



Chapter 4 

DECISION-MAKING LEVELS AND UTILIH ANALYSIS 



INTRODUCTION 

Burt I sumnuurized findingi in the literature related to lelection of training methb Is 
and media and proposed a combined methods-media sdection inrooedure based upon 
teadiinfipoint diaracteristics and a new ^pproadi to media definition and clasdficaticn. 

\Plsrt II summarises findings in the literature rdated to sdection cost*effect\v ^ 
training ivstems» Indudtng-but not limited to-fdection of the most cost-effective 
methodsnnedia combinattons. Fart II also furesents problems and proposes poisible 
solutions pertaining to devdopment of adequate procedures and techniques for utility 
analyds of Amy training systems, whidi indude analyses of training costSt training 
benefits, and training effectiveness* 

To aid undentandii4( of the basic costing requirements of adequate utility analyses, 
attention is concentrated on the two major tnUning-cost categories: devdopmentd costs 
and operational costs. Other cost fscton that appropriatdy should be indudea in utility 
andyses to resolve specific problems, sudi as nonlinear depreciation sdiedulesi oppor* 
tunity costs, and dianging dollar values, are discussed by Fisher (1). 



ARMY DECISION-MAKING LEVELS 

Complete utility andyses of Amy qfstemt must answer questions raised on four 
decision-making levels. 

I* The One>Lesson or OneSubject-Matter Levd 
Questions to be answered indude: 

1. Whidi metiiodsHMdia combinations can effectivdy present and s^^ist 
the leaminr of teaching points? 

2. Are the media available? 

3. Can the instructor use the methods and media? 

4. What differences in payoff and costs would result if one method-media 
combination is substituted for another? 

IL The Course Level 

Basic concerns are the same as in Level I, but broader in scope, and both 
available media and media that must be developed may be considered. Questions to be 
answered indude: 

1. What differences in overall course payofib and costs would result from 
alternative methods4nedia combinations (induding media that may be 
develq[>ed)? 

2. How can on-hand and available media be allocated for the most 
cost-effective r^ults? 
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ni. The Anny Subsystem Level (e^^ combat systons and oxdnance) 

Basic concerns are general short* and long-term training policies and invest* 
ments in new training systems and facilities* Questions to be answered include: 

1. Is it more cost-effective to provide formal or on-the-job training? 

2. In certain manpower areas, would it be more cost-effective to develop 
job aids in lieu of training? 

3. What cost-effective dianges can be made in training in relation to the 
Volunteer Army? 

IV. The Overall Army System Level 

This is the broadest decisionHtnaking levd iiAiere overall policies are gener* 
ated to guide training decisions of all sorts on Level III* IVaining policies at this level 
would relate to the long-term readiness of the Army, and questions may include: 



1. What new instructional ^sterns will be required to meet the needs of 
advancing Army tedmology? 

2. What investments in training are needed now to meet ^aiexgeacy 
manpower requirements without resorting to more costiy ciaA4iaining 
programs? 



The following definitions of tedmical -terms are deseed to aid understanding of the 
remainder of this report. 



DEFINITION OF TERMS 



Benefits of traimng 



Attainment of an oicantzation*s objectiires (benefits 
sought by organizational function) and reduction of 
the or|EanizaUon*s operating cost whidi can be attrib- 
uted to training; such benefits are not training (ef fee- 
ttvenen) results perse, such as student achievement, 
but benefits gained from the extstenoe and function 
of trained manpower. 



Benefit analysis 



A process for determining or estimating the dollar 
values of benefits gained from training. 



Cost analysis 



A process for determining or estimating the dollar 
cost of training. 



Co5t4>enefit analysis 



A process for evaluating the benefits gained from a 
training sytlm in relation to the system's cost, and 
for conqMuring the cost-benefits of altenutive ^^ms 
or of variations of the same system. 



Cost<enectiveness analysis 



A process for evaluating the e^ectiveness of a training 
system (usually in terms of student achievement) in 
relation to the system's cost, and for comparing the 
cost-effectiveness of alternative S3fstems or of variations 
of the same system. 



Effiectiveness of training 



A measure of the usefulness (utility) of a training 
system in attaining the training objectives. 



Outcomes of training 



Results of training that cannot be measured or to 
which numerical values have not been assigned. 



Payoffs of training 



IVaining outcomes that are measured by some numerical 
scale, such as student grades or numbers of dollars. 



System 



A training system, unless otherwise stated. 



TV^dning system Instructional metliods and media, and non-media 

equipment, facilities, persons, and supplies used 
to evoke and control learning activities that are 
necessary for students to attain speciHc training 
objectives. 

Utililv analysis A term which combines cost-beneHt and cost- 

effectiveness analyses. 

Utility of an organization roi organization's usefulness in gaining desired beneHts. 

Utility of a training system A system's usefulness in providing manpower the 

organization can use to gain beneHts. 



UTILITY ANALYSIS: PURPOSE AND EXPANDED DEFINITION 

All traim ' systems are intended to produce desirable outcomes, and training 
officials usuaU:^ are concerned with the effectiveness and costs of the systems. They 
^ould be concerned with idastifying the system design among alternative designs that 
will yield the hi^est degree of payoff at the lowest possible cost. 

Successful identification of tiie most cost-effective system d^ends on th^ quality of 
judgments on the four levek of decision making defined earlier. The quality of judgments 
laigely depends on the validity and reliabihty of the infonnation available to decision 
makers, and the vaUdity and reliability of this information depends, in turn, on the 
adequacy of training-utility analyses. 

Utility analysis includes cost, benefit/ effectiveness, cost-benefit and cost- 
effectiveness analyses. Cost analysis is the subject of Ctu^terS. Cost-benefit and cost- 
effectiveness analyses are related to the overall problem of utility analysis, and they are 
all discussed in Chapter 6. 

UTILITY ANALYSIS ON THE FOUR DECISION-MAKiNG LEVELS 

Use of the term ^training system'' in relation to utility analysis may suggest to some 
that the process is either too broad or too restrictive for use on all four levels of decision 
making, but its 2q[>plicability on all levek can be readily seen when the term "training 
system'' is appropriately interpreted. 

On Levd I, training system can correctly denote even one lesson within a unit of 
instruction, or a single unit of Instruction within a course. 

On Levd II, a course, itself, can correctly be called a training system. 

On Levd ill, all courses or programs that provkie trained manpower for a given 
Army subsystem can also be called a training system. 

On Level IV, all the training ^stems that serve the Army subsystems can be taken 
together and be called the overall Army training system. 

The only difference between the levels, as far as utility analysis is concerned, is not 
in the sqiplicability of the process but in the complexity of the cost and payoff data that 
must be collected and analyzed. The function of utility analysis on all levek would be 
the same-that is, to .provide decision makers with valkl and reliable alternative cost and 
payoff information, whether it is for selection of methods and media at the lesson or 
course level, or a training policy decision at the overall Army system level. 

Reference for Chapter 4 

1. Fisher, G.H* Cost Contiderationt in System Analysis, Report No. R-790*ASD» The Rand 
Corporation, Santa Monica, California, December 1970. 
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Chapter 5 



TRAiMiNG*COST ACCOUNTING AND ANALYSIS 



THE PROBLEM 

If training-cost analysis is to be used for comparing costs of alternative systems or 
alternative system components, such as different method-media combinations, it must 
include the techniques necessary to determine the following: 

(1) The cost of an existing system if its redesign is proposed. 

(2) The changes in cost that would result from redesign of an existing system* 

(3) The costs of proposed alternative systems that have not been implemented* 
Merely adding up the costs associated with different aspects^of an existing system or 

of a single proposed system serves no other purpose than to show how much money 
should be budgeted to continue operationi^of a system or to develop a new one. This is a 
simple cost-accounting function. 

If, however, the costs of alternative systems must be derived and analyzed, the 
cost-accounting techniques must be much more complex and tasks of cost analysts 
become more difficult. As Fisher (1) pointed out, the cost analysts must: 

(1) Account for all sources of training cost, not just those that are obvious or 
that become readily apparent. 

(2) Identify the cost sources that, when they vary, contribute most to changes 
in system costs.-- 

(3) Determine and compare the magnitudes of cost changes and present the 
results to decision makers. 

T.< \ responsibilities may seem fakly simple when stated in this way, but the 
problems rest in the necessities of: 

(1) Providing the cost-accounting structures that will permit identification of 
all sources of training costs and division of these sources into subcategories 
that can be manipulated for meaningful cost comparisons. 

(2) Selecting or deriving techniques for determining which cost sources 
contribute most to changes in system costs. 

(3) Selecting or devising feasible means for determining or estimating magni- 
tudes of cost changes. 

(4) Selecting or deriving mathematical formulas for use in meaningful compar- 
isons of cost differences in alternative systems. 

The problem of deriving adequate techniques and procedures for Army training cost 
analysis involves identification of those costs that exist that may be applied to or be 
adapted for appUcation to Army conditions and requirements, and development of new 
ones that may be needed to solve unique Army problems. 

A thorough review of the literature reveled that although a number of cost analysis 
models have been developed, none are adequate for Army use. Some, however, have 
features that can be used or that can be adapted for use, and these will be discussed in 
the sections that follow. 
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C»STACCOUNTING STRUCTURES 



We agree with Kemp (2) that a useful instructional cost-accounting structure should 
have two major cat^ories by which all sources of training cost can be classified: 
(a) developmental costs (basic expenditures that must be made before training can occur), 
and (b) operational costs (expenditures necessary for the conduct of training). 

Kemp also provided the following examples of cost sources that should be included 
in each major cat^ory. 



Developmental Costs 
Planning time. 



Staff time. 



Percentage of salary for time spent by each member 
of the planning team on the project (or number of 
hours spent by each member multiplied by his hourly 
salary rate), and fees for consultants. 

Percentage of salary for time spent by each staff 

member engaged in planning and production, and in 
gathering materials (or the number of hours spent 
by each person multiplied by his houriy salary rat^). 

Supplies and materials . * 

Outside services . Services necessary for preparing or purchasing 

materials. 



Construction or 
renovation of 
facilities . 

Equipment. 

Installation of 
equipment. 

Testing, evaluation , 
revision, 
reproduction. 

In-service education. 



Overhead costs. 



Miscellaneous. 



These include personnel time and cosCs during the 
system validation phase. 



Operational Costs 

Administrative 
salaries . 

Faculty salaries. 



Other salaries. 

Replacement of co n* 
sumable >7Ia 
damaged materials. 

Repair of damaged 
equipment. 

Depreciation of 
equipment. 



Training for teachers, aides, and others who will 
participate in the program during implementation 
(cost for time). 

Expenditures such as utilities, telephone, furniture 
and room or building costs, or depreciation allowances, 
incurred during the developmental phase. 

Travel and other items. 



Salaries (based iipon percentage of time) chargeable 
to the instructional system. 

Salaries for time spent in the program-forking with 
groups and individual students, planning daily 
activities, evaluating programs, revbing activities 
and materials. 

Salaries for aides, maintenance technicians, and others. 



Overhead , Expenditures such as those for utilities, telephone, 

facilities, furnishings, custodial services 

Evaluating and 

updating nmterials. 

The overall i^plicability of Kemp's list of cost sources to Army training must be 
determined' by further study, but student maintenance, at least, should be added. When 
all Army cost sources are known, the costs of developing and operating a given training 
system can be established by calculating the cosi^ related to each source and summing 
the amounts. This was done, in part, in a HumRRO study by JoUey and Caro (3) to 
compare selected costs of Army flight training and synthetic flight training. A similar 
HumRRO study was made by Cato et ai (4) for the Coast Guard. But this is not enough 
for our purposes. 

To determine how various changes in systems design would effect changes in system 
costs, Kemp's cost-accounting structure must be refined so that cost &ctors can be 
specifically identified. This means that the training*time foctor must be added and that 
various cost sources must be related in meaningful ways, such as students and instructors, 
so that various' student*to*instructor ratios can be calculated. 

An example of how a refined Army cost-accounting structure mi^^t appear is shown 
in Tables. This example is merely illustrative, and it is not intended to show all the 
kinds of cost factors that might ultimately emerge. 

An adequate cost-accounting structure must also provide for classification of cost 
sources according to the various functions involved in developing, operating, and evalu- 
ating a training system. At this point, seven training-function categories have been 
identified: 

(1) Instruction. Expenditures connected directly with instruction. 

(2) Practice. Expenditures connected witti students practicing during the 
course. 

(3) Non-instruction . Expenditures connected with student activities not a part 
of the instruction, such as testing^ field trips, etc. 

(4) Administration and Services . Expenditures for administration and other 
services. 

(5) Operation . Expenditures for training facilities and utilities. 

(6) Maintenance. Expenditures for maintenance of facilities, equipment, and 
students. 

' (7) Pre- and ^Post-Instruction. Planning, producing, evaluating, and other activ- 
ities related to instruction, but not part of it. 

Gassifying cost sources according to functions is highly important so that specific 
changes in cost can be readily identified when changes are proposed in any of the 
functions. Therefore, identification of all Army cost sources which should be related to 
each function would be one of the first major research tasks. 

Additional devices to aid cost accounting wouk) merely be forms to facilitate the 
analyst's work. One form would be for cat^orizing changes in cost according to the 
previous training and training-related functions. Such a form would permit easy 
summation of costs in each category for each alternative system so that totals for each 
system could be easily compared. 

Other forms would help analysts in calculating cost factors, such as student- 
instructor ratios, various costs per student, and costs per student-hour of instruction. 

MATHEMATICAL FORMULAS FOR COMPARING SYSTEMS COSTS 

Regardless of the cost foctors and factor manipulations that may occur in designing 
alternative systems, the resulting total costs for eadi system will still be identified as 
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Tables 

Example of a Refined Cost-Accounting Structure* 



Inttructional Media 
Kind$ 

Number used 
Percentage of time used 
Investments 

Planning 

Acquisition 

Rental 

Production 
Costs per student 
Costs per student*hour 

Inttructional Materials 

Investments 

Planning 

Acquisition 

Rentals' 

Production 
Consumption per student 
Costs per student 
Costs per student-hour 

Instructional Fadlitiet 

Classrooms 
Other space 
Percentage of time used 
Cost per student 
Cost per student*hour 
Peak use percentage 

Inttructional Equipment 

Total investment 
Investment per student 
Investment per student-hour 
Useful life 

Depreciation cost per student 

and per student«hour 
Percentage of time used 



Instructional Function 

Number of instructors 
Number of students 
Average instructor hours per day 
Instructor cost per student 
Instructor cost per student-hour 
Student-instructor ratios 
Total number of hours 
Hours per student 

Practice Function 

Number of instructors 
Number of students 
Average instructor hours per day 
Instructor cost per student 
' I nstructor cost per student-hour 
. Student-instructor ratios 
Total number of hours 
Hours per student 

Administration 

Personnel 
Facilities 
Space 

Cost per student 
Cost per student-hour 

Maintenance and Janitorial 

Personnel 
Facilities 
Space 

Supplies ^ 
Cost per student 
Cost per student-hour 

Miscellaneous 

Office supplies 
School supplies 
Ammunition 
Utilities 
Other 

Cost per student 
Cost per student-hour 



•The exact lilt of factors will depend upon the cost'tneiysis problem to 
be solved* That \%, for each specific problem, sonrw factors may be replaced by 
others* 



developmental costs and operational costs. The^tWft^categories alone are enou^ for 
deriving mathematical formulas that can be used i to meaningfully compare differences in 
system costs. Examples of mathematical formulas and their uses are given in Appendix C. 

One mathematical formula, with sli^t variations, would yield training costs in terms 
of various training ou^ut units, sudi as cost per group of students trained in each 
replication of a course, cost per student, and cost per student-hour of instruction . 

This formula, when used with each replication of a course, would be used to 
calculate changes in ouput-unit costs. When these changes for alternative systems are 
plotted in a graph, their relationships can be easily compared. 

Another mathematical formula permits prediction of the time when the unit costs of 
two different systems would be equcd. 

A third formula permits calculation of cumulative unit costs (resulting from course 
ireplications) to yield alternative cost relationships that can be easily compared. 

To avoid possible confusion in the following section, it must be emphasized that 
these mathematical formulas, or formulas similar to them, will be used for the com- 
parison of alternative training systems, r^ardless of the techniques that are used to 
determine how system costs diange when system designs change. In other words, the 
techniques discussed in the following section are not intended to replace these formulas; 
they are merely more sophisticated cost-accounting techniques for determining system 
costs and cost differences that should be compared. 



COST ANALYSIS MODELS IN THE LITERATURE 

To this point we have discussed the cost-accounting aspect of training cost analysis, 
identified the cost-accounting structure that must be used for computing output-unit 
costs, and provided (in Appendix C) mathematical formulas for calculating output-unit 
costs and graphical techniques for displaying the results of these calculations. 

Attention must now be given to existing cost-analysis models to identify other 
problems not yet considered and to note additional techniques that may be helpful in 
deriving a model or models to meet Army requirements. Four elaborate models are 
summarized in Appendix D. These and other models are cited in the foUowing 
paragraphs. 



USEFUL TECHNIQUES 

To facilitate the design of alternative training systems, cost analysts need additional 
techniques for predicting how much training-unit costs (such as cost per student-hour of 
instruction) will be affected when certain changes are made in a system *s design. 

If cost analysts can identify the cost sources that contribute most -to changes in 
output-unit costs, systems engineers could then concentrate on thos^ cost sources as the 
most important to consider in systems design. This is not easily done without the special 
techniques provided in the COST-ED model in Appendix D. T^ese techniques are useful 
in determining: 

(1) The interaction effects of cost factors, which show how changes in one 
cost factor effect changes in another cost factor. For example, this technique answers 
questions such as these: If more intensive use of training facilities is planned, how much 
will t^is affect the cost per student4iour of instruction? Or, if programed instruction is 
planned to reduce the number of instructors, how much will this diange the cost of 
instructors and output-unit costs? 
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(2) Senativity of output-unit cost to changes in cost factois, which shows the 
degree of percenb^e dianges in ou^>ut-unit costs caused by changes in different cost 
factors. For example, a diange of 10% in on' cost factor might result in only a 1% 
change in unit cost, whereas a change of 5% in another cost factor might result in a 10% 
change in the unit cost. 

These techniques can be used by the Army on all four levels of training decision 
making. Whether other features of the computerized COST-ED model would be useful to 
the Army would have to be determined by further study. 



PROBLEMS IN DETERMINING COSTS 

The usefulness of other existing cost-analysis models for development of an adequate 
Army model, or models, is found chiefly in their techniques for estimating costs. The 
problem of which cost sources should be considered is different on each level of decision 
making. In other words, the specific costing problems are determined by the purpose in 
performing cost analyses, which is different on each level. ' 

At the lowest level, the costing problem is simplest because the purpose is to 
compare only the alternative costs of specific methods^nedia combinations within a 
narrow range of system design. But even at this level, one cannot always determine exact 
dollar costs, especially when production of media must be conddered. As Fisher (1) 
correctly noted, dollar^ost values usually do not represent absolute costs; rather, doUi^ 
are conveniently used to represent time and cost sources. 

As one proceeds to the higher levels of decision making. Fisher's statement becomes 
even more true, and it becomes increasingly important for cost analysts to provkle for 
tradeoffs that can be considered to keep instructional costs within budget restrictions 
(Craig and Dietrich, 5). 

Also, at hif^er levels the cost factors to be conddered become fewer. A study done 
for the Air Force by Westinghouse Learning Corporation (Appendix D) provides an 
example. In analyzing the problem of selecting service-wkle media for Air Force training 
(a problem for ttie highest level of decision making), only the costs of training materials 
production, media equipment, facilities, sun[>lies, and personnel expenses were con^ered. 

The Westinghouse researchers found that for purposes of comparing different system 
costs, they needed only the critical ones, not all possible costs. They ^also found that they 
needed only a useful order of cost magnitude, rather than accurate totals. 

Sovergin (Appendix D), in a three-volume report prepared for the U.S. Office of 
Education, drew conclusions that generally support the Westin^ouse approach. 

Speagle (Appendix D) listed only eight cost foctors considered in his study of three 
major media systems (TV, computer access, and computer-assisted instruction). These 
factors were: 

(1) Heavy, inherent overhead and fixed costs 

(2) Cost-saving technologies 

(3) Geographical concentration of the student population 

(4) Cooperation among sdiool districts and systems 

(5) Machinery for evaluating the quality of and effectiveness of teaching 
techniques and materials 

(6) Levels and types of teaching programs desired 

(7) Rate of learning under innovative techniques 

(8) Possibilities of replacing traditional teaching with instructional technology 

Speagle was concerned with cost analyses on a hi{^ decision^naking level, so the 
cost factors considered were gross. For decisions on lower levels, the cost accounting 



43 



must be more detailed and precise. Gardner (6) concluded after surveying the cost* 
analysis literature, that a way is needed to avoid arbitrary assignment of costs to 
instructional functions. This means» he said, that there must be minute accounting of 
resource usage. He also observed that, university faculties and administrators, with whom 
he was concerned, appear unwilling to accept the cost-accounting requirements. The 
degree to which these problems exist for the Army has yet to be determined. 

The refined cost*accounting structure we proposed earlier in this chapter will greatly"^ 
help in identifying specific training costs, particularly in the lower levels of decision 
making, but the overall Army policy level (Level IV) will continue to be fraught with 
nmny cost-uncertainty problems. All existing models designed for high-level cost analyses 
leave judgments of cost uncertainties largely to intuition, yAiich can result in grossly 
inaccurate cost escimates. 

In an effort to supj)Iant intuitive cost judgments, Schafer (7) dmonstrated that 
statistical analyses can provide cost probabilities for use in making1i)ett^ cost estimates. 
These techniques, combined with cost considerations found in existing high-level models, 
may provide the Army with the means of making the best possible cost estimates. 
Schafer*s statistical models may also be quite helpful on the second and third levels in 
estimating proposed training costs. 

The models summarized in Appendix D also provkle other specific information that 
may be used for Army purposes, particularly in considering large media installations. 

In addition, Rapp et al. (8) have provided a mode! for evaluating specific costs of 
alternative course designs which may be useful in analyzing Army course costs on 
Level IL Although this model was developed for evaluating costs of reading and arith- 
metic programs for junior high schools, application of its features for Army use should be 
studied. 

Although numerous other cost studies were found in the literature, few seem useful 
as sources of information that mi^t akl development of adequate Army cost-analysis 
models. 
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Chapter 6 
TRAINING-UTILITY ANALYSIS 



The purpose of training-utility analysis is to provide decision makers with valid and 
reliable information for use in judging which alternative training c^stem will yield the 
greatest payoff at the low^ possible cost, 'nradning-utility analysis can be used to 
determine which system: 

(1) Will cost least if all systems considered will yield the same payotf. 

(2) Will yield the greatest payoff if costs of all the systems will be equal. 

(3) Will yield the greatest payoff at the least cost. 

These objectives of utility analysis will be discussed in the following sections in 
relation to specific problems. 



PROBLEMS OF TRAINING-UTILITY ANALYSIS 

As explained in Chiqpter 4, training-utility analysis includes five kinds of analyses: 
(a) cost; (b) benefit; (c) cost-benefit; (d) effectiveness; and (e) cost^effectiveness. Cost 
analysis axiA related problems were discu!»ed in Chapter 5; the remaining kinds and their 
related problems are treated in the following paragraphs. 



BENEFIT ANALYSIS 

The benefit gained firom adequate military organizations is national security, and this 
benefit consists of two sub-benefits: (a) detening enemy attack, which is gained firom the 
existence of adequately trained and equipped manpower^ and (b) repulsing enemy attack 
when it occurs, which is gained from military action. 

Eadi of these benefits can be subdivkled into more specific benefits, such as 
deteixing and r^ulsing attack on the home country and deterring and repulsing attack on 
allied countries, and so forth. 

Anoth^ benefit often derived firom training is reduction of the organization's 
operating cost. 

The problems of benefit analysis are of two major kinds: 

(1) Identification of specific benefits or portions of benefits that can be 
attributed to training. 

(2) Determination or estimation of the dollar values of these benefits. 
Identification and evaluation of reduced Army opemting costs that can be attributed 

to training are less complicated than the same tasks in relation to national security, but 
they are still difficult. 

In thinking of Army readiness, suppose that equipment down-time is a problem. 
Someone must determine whether excessive down-time is due to inadequately trained 
maintenance personnel or due to other factors, sudi as inadequate supervision of 
personnel or lack of spare parts. If down-time can be reduced by more adequate training, 
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the operating costs that can be saved may be estimated by con^ering the following 
foctors: 

(1) Cost of parts lost because of faulty preventive maintenance. 

(2) Cost of personnel time wasted because of down*time. 

(3) Other excessive operating costs due to down-time. 

(4) The percentage of the total that probably can be saved by training. 

If the costs that can be saved are greater than the cost of more adequate training, 
the reduction in operating cost may be justification enou^ for training improvement. If 
not, the, probability of losing national security benefits should be considered. 

What might be the cost consequences of inadequate training if lack of readiness 
reduces the capability of repulsing sftlack? Or in time of war, what might be the cost 
consequences if battles or attack opportunities are lost? 

Obviously, it woukl be very difficult to estimate dollar costs of sudi consequences, 
but despite the problems, a purist woukl say it must be done. Actually, someone does 
indirectly estimate the value of national security vihen military funding is being con- 
sxlered. This is necessary to justify q>propriations, althou^ the estimates may not be 
very accurate; And for training-utility analysis, as Quade(l) pdnted out, dollar 
evaluation of benefits does not need to be highly accurate; the benefit-value and 
training-cost estimates must indicate only the relative merits of alternative training 
systems. 

Quade compared these requirements to those encountered in cost-benefit studies of 
the proposed supersonic transport. He observed that no one could confidently predk:t the 
benefits of SSTs or determine whether demand would be great enou|^ for their com- 
mercial success, but evaluations of competing designs were made anyway by con^aring 
their relative merits. 



COST-BENEFIT ANALYSIS 

TVaining cost-benefit analysis involves evaluation of national security benefits and 
reduction of Army operating costs (whu:h can be attributed to training) in relation to 
training costs. Analyses involving national security benefits probably would i performed 
on the fourth decision-making level, while analyses involving reduction of operating costs 
may be performed on both the third and fourth levels. 

To emphasize the intended value of training-utility analysis, two aspects of cost- 
benefit analysis shouM be consklered. One is merely estimating the utility value of a 
given training system. The other is comparing the utility values of alternative training 
systems to identify the most cost-effective system. 

The process for estimating the utility value of a given training system is relatively 
simple. For example, if reasonable- dollar-value estimates of national security training 
benefits can be made, the utility value of the existing overall Army training system can 
be estimated by subtracting the dollar cost of the system from the estimated dollar value 
of the benefits gained. The same couki be done to estimate the utility values of training 
systems on the Army subsystem level. This kind of analysis alone is not helpful, however, 
in making choices between alternative systems. 

Therefore, in addition to training-cost accounting and benefit analysis, training- 
utility analysis must include appropriate cost-analysis procedures so that the relative 
cost-benefit (utility) values of alternative trai ning systems can be readily estimated and 
compared. P 
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EFFECTIVENESS ANALYSIS 



IVaining^effectiveness analysis is the process of measuring training outcomes to 
convert them to training payofb. 'Araining outcomes of interest may be tangible or 
intangible* For example, we may consider student job or jobHrelated performance as 
tangible and student attitudes and n^orale as intangible. Whether tangible or not, every 
training outcome to be used in cost-effectiveness analysis must be measured. This is eaiy 
to state, but valid and reliable measurement of outcomes to obtain meaningful payofb is 
difficult, and the intangible ones are more difficult to measure than the tangible. 

Since the puipose ut training is to prepare personnel for job perfonnance, the most 
meaningful payoffs are those obtained by reliable jobi>erformance ^>praisals. But reliable 
job-performance iq[>prai8als are difficult to obtain, so this kind of feedback from the field 
to schools is rare. For this reason, training decision makers usually base their judpnents 
on data obtained by tests whidi are administered on the course level, 

Sudi tests are intended to reliably measure student attainment of training objectives, 
and the test scores are usad as measures of training effectiveness. However, for test results 
to be valid as effectiveness measures, they must reliably predict student job performance. 
To do so, the following conditions must hoU: 

(1) The training objectives and standards must be based upon job-entry per- 
formance requirements and standards* 

(2) The tests must be based directly upon the training objectives and standards, 

(3) The number of test items must be laige enough to reliably represent the 
objectives and standards. 

(4) The tests must be reliably administered. 

To determine whether tests are valid predictors of job performance, the perform* 
ances of students on tests must be compared with the job-performance evaluations of the 
same students. If the results of these comparisons diow tiiat test scores do predict job 
performance, then the tests can be considered valid, and this vaUdity can be taken as 
evklence that the test scores can be used meaningfully as training payoffs. 

Before proceeding, it must be noted that some authors connder redr.ced instruc* 
tional time as a payoff, Althouf^ reduction of training time is often desired, it is a cost 
factor, not an effectiveness (payoff) factor, and training time must be taken into account 
as cost in cost-effectiveness analysis, ^ 

COST*EFFECTIVENESS ANALYSIS ' 

'nraining payoffs are used in cost*effectiveness analyses to determine or estimate 
training-utility values and to calculate cost-effectiveness indexes. 

If training payoffs are to be used for deriving training*utility values, they must have 
dollar values assigned to them. This does not mean that calculations cannot be done with 
non-dollar*valued payoffs, but it does mean that results of the calculations cannot be 
I»roperiy called 'hitility values''; instead, they should be called *^cost«effectweness 
indexes.*' 

Cost-effectiveness indexes can be very useful as indicators of utility values. For 
example, suppose that Army officials are concerned about the malfunction rate in a 
weapons system and want to compare alternative training systems for reducing malfunc- 
tions. To get utility values as bases for comparing the systems, someone would have to 
assign dollar values to those malfunctions which each training system might eliminate. 
Usually, however, the training-utility analyst need not be concerned. He can assume that 
someone may assign dollar values if they want that kind of information. So he wouk! 
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calculate a cost^fectiveness index by using non-doUar-valued payoffs, sudi as valid test 
scores. 

The problem of predicting future training payoffs for comparing systems remains, 
but it can be solved. Suppose, for example, that an existing course has been in operation 
for some time and that student test scores have been kept on record* Suppose also that 
the number of students that have been trained is a valid and statistically reliable sample 
of the student population yet to be trained. The test secures could then be used to predict 
the proportions of future students tdio would &11 into qpedfied studenti>erformance 
categories. 

If the design of this course is to be dianged to make it more effective, the new 
course must be devdoped and tested with a sample of students large enough to validly 
and reliably rq[>resent the future student peculation. The test scores fircHn this group 
could then be u^d to predict tiie proportions of future students who woukl fall into 
specified student performance categories. 

The results of these two sets of figuies mi^t appear u shown betow: 

Qradt'Ltvtl Cs^tyoHo 

L !L UL 

Old eourw .20 .25 .55 

Newcoune .55 .25 .20 

If the test standard is '"Go, No-Go** the categories mi^t appear as below: 

Go NO'Oo 

Ok: count .75 .25 

New course .95 .05 

If all students were traUied to the same standard, such as ''Go,** effectiveness of the 
alternative courses would not vary, but training time probably would, and time refHre- 
sented in cost would still yield a difference between the cost-effectiveness indexes. 

If intangible training outcomes, sudi as attitudes and morale, can be measured in 
some meaningful way, then they could become training payoff^, and the test scores could 
also be used in the way just described. It must be emi^asized again, however, that tests 
used for obtaining training payoffs must be both valid and reliable, otherwise they would 
yieM spurious data that could seriously reduce the validity of trainingnitility analysis and 
thus lower the quality of decision-maker judgments. 



A MATHEMATICAL F0R:VIULA FOR COST-EFFECTIVENESS ANALYSIS 

Assuming that reliable cost estimates and valid effectiveness data are available tot 
two alternative traini)ig systems, a mathenuitical formula can be ai^lied to calculate the 
systems* cost-effectiv^mess indexes. If the effectiveness data are dollar values, then the 
systems* utility values could be calculated. 

To demonstrate how cost-effectiveness indexes can be used to compare different 
training systems, a formula wtis derived for use on the course Lvel. This formula is given 
in Appendix E where its use is demonstrated with actual training cost and test data. 

Derivation of other formulas diould await determination of the kinds of problems 
the Army wishes to treat in utility analyses. 
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TRAINING4ITILITY ANALYSIS IN GENERAL 

Our limited knowledge of how uUKty analysis can be applied to solve Army trainii« 
problems may have caused us to overemphasize use of the techniques on the course leveJ. 
To Aow some implications for the process on all four Army decision-making levds, we 
have posed sample quesUons for each level that couki be answered. These questions are 
ghren in Table 9. At this Ume, the questions must necessarily be general; only further 
study can determine the 4>ecific ways in whidi the Army can use the te^Ankiues. 



LITERATURE PERTAINING TO UTILITY ANALYSIS 

A thorouf^ seardi of th^ literature pertaining to utility analysis revealed little 
information of 4>ecific value for Army purposes. The chief reason for this lack is that 
most of the literature pertains to public educatton problems that are not specifically 

Tsb!s9 

SampIt TfMkuoQ Quastiom 



Ltvtt 



OuMtiont 



i. One Iftion or one 
subjtct*instttr Isvtl 



ii. Courstltvtl 



iii. Army tubiysttm 
Itvel 



IV. Overs!! Army 
tystsm Itvtl 



Whirh inttructionil procsdurt p^ iucti htghtr student ptrformsncs? 

Which mtthod-mtdis combination will rtduct training timt? 

Which inttructionilHJnit design %rill yield s lo^ rst^ of student 
failures? 

Towsrd which procedure do instructors and students have more 
positive sttiti^? 

Which course design will produce desired student performsnce in the 
least time? 

WWch desigr. yields the highest rate of high-performing students? 

Which design will enable low*«bility students to achieve higher levels 
of performance? 

Which training system wit! produce higher Job performance? 

Which system ySckH a higher rate zf d<illed*personnel? 

What effects will on*^1)e^ training have on iob perfornunce? 

Which is the overall Irast-costly wsy of prepering p*.sonnel for their 
iobs? 

Which training policies are best for achieving defense«system readiness? 

What long-term investments should be made in trsining? 

Which training policy should be adopted for nrieetirii future mili"^ ry 
needs? 

What impiic^' ^ do new technologies (weapon systems) have for 
Army tralnii < 

Which training . licies will be most effective for meeting state-of- 
emergency needs? 
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related to Army training. Of the few reports in the area of vocational training, rcost deal 
with interests of profit-making organizations that also have, little in common with Army 
problems. 

Nevertheless^ some of the literature (cited in the foUowing sections) may be helpfiil 
in tailoring training*utility analysis for Army needs. 



QUESTIONS POSED 

One future benefit found in the literature condsts of questions raised and discussed 
which must be answered in relation to Am^ needs. Luinsdaine(2) asked three crucial 
questions and pointed out that they have differaat meanii^ on different levels of 
decision making. The questions are: 

(1 ) . What are instructional outcomes? (What outcomes are of interest)? 

(2) What constitutes worthirtifle differences in (Instructional) effectiveness? 

(3) How can differences in outcomes be measurod? 

Miller (3) poses other questions that msy also be asked by Army training officials, 
such as: 

(1) What can I buy witii a smaU increment in cost? 

(2) What is tiie critical mass necessary to produce (instructional) results? 
(referring to the number of designers and producers of instructional 
systems) 

(3) Can I find others to share tiie cost? 



COST-EFFECTIVENESS MODELS 

Various cost-effectivraess models have-beafi presented in the literature for ^>ecific 
2q[>plications» some of iH^idi may be adapted for Army use. 

Alkin (4) outlines a modd that permits deci^on makers to compare outcomes that 
are measured in different units. His model is for evaluation of public school inrograms on 
five levels of financial input. The problems involved may be somevdiat analogous to 
Army problems on the third and fourth levels. 

Kiesling (5) att^pts to relate problems of evaluatii^ educational outcomes to those 
of evaluating government ou^uts in general^ and his analytical procedure for determining 
costs of media and their place in an instructional system may be useful. 

Tanner (6) developed ''an ejqpected opportunity loss** model that may deserve 
serious attention. His decision^naking tedmique uses subjective data» whidh are rankings 
of courses according to their eiq>ected contribution to the total prc^ram. 

If the Army diould conskler use of Computer-Asdsted Instruction (CAI)» the 
cost-effectiveness study by Kopstein and Seidd (7)» vriiidi ccmipared CAI ^stems and 
traditional instruction^ probably would be useful. 



MATHEMATICAL FORMULAS 

Two sources probably will be h^ful in deriving additional cost-effectiveness mathe- 
matical formulas that may be needed for specific Army purposes. These are Cronbadi 
and Gleser(8) and Edwards (9). Both npply deciston theory in developing mathematical 
formulas that may be adapted for special Army cases. 
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OTHER CONSIDERATIONS 



Of growing intetest in general education is the concq>t of 'Accountability*' of 
teachers and officials for ttie results of education* The concept arose from tiie experi- 
ments of some public sdiools withl)erformance contracting. This term refers to contracts 
made by private concerns witii public schools to instruct students, which provide that the 
contractors be paid according to the degree of student achievement. In effect these 
contractors guarantee student adiievement or they do not get paid. Hie notion that 
regular teadiers and school officials should likewise or somehow be hdd accountable for 
student ac^evonent has elicited wideq[>read concern. The concept has also grown to 
include educator reqx>nsibility for decisions made in expending funds and for the value 
of educational results. 

Concern about accountability has resulted in numerous ptiblications which pose 
questions, discuss problems, and propose means by jvfaidi accountability can be achieved. 
The most important of these publications are Lennon (10), StufQebeam (11), Earro (12), 
Wyime(13), and Roberson (14). hi addition, two journals^ devoted entire Issaes to 
accountability. 

Whether ttie Army dio' *i be interested in accountability or in ^performance con- 
tracting are moot questions. 

Kirby (15), in estimatii4 the costs aiMl benefits for both individuals and the Federal 
government in ttie IVaining and Tedmology Project, derived a *^te of return** analysis 
that ttie Army may want to conader in relation to training behefits, particularly in 
relation to volunteer Army concepts. 

Other sources may idso contain implications for Army training. Sands (Ig) provides 
a model for recruiting, selecting, inducting, and training sufficient personnel to meet 
qpedfied quotas, and a wide range of maiq[K>wer development problems is analyzed and 
discussed in a book by V^mcen (17) . A study by the Air Force (McCall and Wallace, 18) 
to determine the d^ree to whidi Air Force training is transferable to civilian jobs may 
also be pertinent. 
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Appendix A 



METHODS-MEDIA SELECTION 
PROCEDURES IN THE LITERATURE 



Gagne (1) provided a matrix (Table A-1) for selecting media according to the degree 
(^^Yes/' **No»** ^^iMted^') those media can perform the instruction^ functions discussed in 
Chapter 3. The function categories are too gross and the media named are too few to be of 
much assistance to the ^stems engineer in precise selection of media for specific purposes. 
This criticism more or less applies to other media selection schemes presented in this appendix.. 



Table A*1 

Instructional Functions of Various Media* 





Mtdia 


Obiecti; 
Demon- 
ftration 


Orat 

Communi- 
ntion 








■ — 
Sound 
Moviit 




Function 


Printtd 


Still 
Plcturtt 


Moving 
Pictures 


TMching 
Machines 


Presenting the 
















stimulus 


Yes 


Limited 


Limited 


Yes 


Yes 


Yes 


Yes 


Directing attention 
















and other activity 


No 


Yes 


Yes 


No 


No 


Yes 


Yes 


Providing a model 
















of expected 
















performance 


Limited 


Yes 


Yes 


Limited 


Limited 


Yes 


Yes 


Furnishing external 
















prompts 


Limited 


Yes 


Yes 


Limited 


Limited 


Yes 


Yes 


Guiding thinking 


No 


Yes 


Yes 


No 


No 


Yes 


Yes 


Inducing transfer 


Limited 


Yes 


Limited 


Limited 


Limited 


Limited 


Limited 


Assessing attain- 
















ments 


No 


Yes 


Yes 


No 


No 


Yes 


Yes 


Providing feedback 


Limited 


Yes 


Yes 


No 


Limited 


Yes 


Yes 



'From Gagne (1^). Permittion to reprint given by copyright holder. 



Allen » et aL (2) related instructional media to learning objectives in the matrix shown in 
Table A*2. Allen also presented two other nmtrices, Tables A*3 and to show equipment/ 
media relationships and considerations^ and to relate media and media characteristics. 
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Table A-2 

Media Related to Learning Objectives* 





Learning Objectives^ 


Typtof 
Inttructioncl Media 


Learning 
Factual 

Information 


Learning 
Visual 
Identifi- 
cations 


Learning 
Principles, 
Concepts, 
and Rules 


Learning 
Procedures 


Performing 

Skilled 
Perceptual- 
Motor Acts 


.Developing 
Desirable 
Attitudes, 
Opinions, St 
Motivations 


Still Pictures 


Medium 


HIGH 


Medium 


Medium 


low 


low 


Motion Pictures 


Medium 


HIGH 


HIGH 


HIGH 


Medium 


Medium 


Television 


Medium 


Medium 


HIGH 


Medium 


low 


Medium 


Training Aids 


low 


HIGH 


Medium 


Medium 


low 


low 


Audio Recordings 


Medium 


low 


low 


Medium 


low 


Medium 


Trainer 














(Simulator) 


Medium 


HIGH 


HIGH 


HIGH 


HIGH 


Medium 


Progranimed 














Instruction 


Medium 


Medium 


Medium 


HIGH 


low 


Mediunn 


Demonstration 


low 


Medium 


low 


HIGH 


Medium 


Medium 


Printed Textbooks 


Medium 


low 


Medium 


Medium 


low 


Medium 


Oral Presentation 


Medium 


low 


Medium 


Medium 


low 


Medium 



*W.H. Alien, R.F. Filep, and S.M. Cooney. Visuatand Awfio Pnsenmlon in Machine Prognnwi 
imtruethn, OE Final Report, 1967,®University of Southern California Reprinted by permission. 

^£ach type of instructional media is rated on one of three levels of effectiveness ("tow," "Medium," 
"HIGH") in obtaining the learning objectives. 



Table A-3 



Equipment/Media Relationships and Considerations* 



Instrument 


Media Used 


I MJ^terials 
Production 
Considerations 


Availability 
of Facilities 
and Equipment 


Equipment Cost 


1 . Filmstrip or slide 35mm f ilmstrips or 
projector ^ 2x2 slides 

2. OverBead trans- Still pictures and 
psrency projector graphic representa- 
tions. 


Inexpensive. May be 
done locally in short 
time. ' 

Very inexpensive. 
May be done locally 
In short time. 


Usually available. 
Requires darkened 
room. 

Available. May be 
projected in light 
room. 


low 
low 


3. Wall charts or 
posters 


Still pictures 


Very Inexpensive. 
May be done locally 
in a very short time. 


Available. No 
special equipment 
nNded. 


very low 


4. Motion pictures 
(projection to 
groups) 


16mm motion 
picture (sound or 
silent) 


Specially-produced. 
Sound film Is costly 
and requires 6-12 
months time. 


Usually available. 
Requires darkened 
classroom. 


moderate to high 



(Continued) 



Table A-3 (Continued) 



Equipment/Media Relationships and Considerations 



instrument 


Media Used 


Production 
Considerations 


Availabiiitv 
Of Facilities 
and Equipmtnt 


Equipment Cost 


5* Motion picture 
projtct fonts 
rtpttitivi loops 
(8mm silint) to 
indrviduals 


8mm motion picture 
film (silent) 


Spaciel production 
normelly necessary. 
May be produced as 
16mm film alone or 
locally at low cost 
and in short time. 


Not normally avail- 
able. Will need to 
be specially procured 
to meet requirement 
of instructional 
program. 


low per unit, but 
moderate for groups 


&Mlgn9tictapt 
recordtr 


X" magnetic tape 


Euy and.inexpensive. 
Usually produced 
locally. 


Available 


low 


7* Rtcord pliyir 


33l/3M5or78 
rpm disk recordings 


Need speciel record* 
ing facilities. Usually 
-commercielly made. 


Usuelly available 


low 


8. Display area 


Treinersand training 
aids 


May vary in complex* 
(ty and in difficulty 
of production. Ccir;* 
Dontnt nartiMtv to 
obtain. 


Usuelly available 


veriesfrom low to 

high 


9. Television (closed- 
circuit) 


Live presentations. 
Motion picture film. 
Videotape recordings. 
Still pictures. 


Normally requires 
large and skilled pro* 
duction staff. 


Not normally 
available 


moderate to high 


10. Teaching machines 
8i programmed 
textbooks 


Programmed 
material 


Some programs avail* 
able commercially. 
But will normally be 
specially prepared for 
course. 


Not normally 
available 


low per unit, but 
moderate for groups 


11. System combina- 
tions 


Television. Motion 
pictures. Still 
pictures. Audio 
recordings. 


Complex. Probably 
will be done locally 
to meet specific 
requirements. 


Not normally 
available 


moderate to high 



•w .H. Alien, R.F. F ilep, and S.M. Cooney. Visual and Audio Prasantation in Machkta Programed Instruction, OE 
Final Report, 1967, © University of Southern California. Reprinted by permission. 
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Appendix B 

MEDIA CLASSIFICATION SCHEMES IN THE LITERATURE 



Meredith (1) attempted to provide an adequate media classification scheme by 
specifying four major categories to which he thought media should be related: 

(1) Physical variables in the material and form of the medium providing 
the stimulus. 

(2) Neuroanatomical variables in the sensory-motor structure involved in 
learner responding. 

(3) Ecological variables related to architecture and other environmental 
factors among which the media would be placed. 

(4) Collective variables that include time, memory, learning, student history, 
attention, purpose, expectation, imagination and anticipation. 

Several somewhat similar attempts were made by Dale (2), Ely (3), Knowlton (4), 
and Lumsdaine (5). The most recent is by Bretz (6). His classification"^scheme, shown in 
Figure B-1, was criticized by Ely (?)♦ " 

Tosti and Ball (8) provided a classification system in three dimenfdons: encoding, 
duration, and response-demand. These are shown in Figures B-2, B-3, and B«4 and are 
used to relate the ^dimensions to each other and to other media and instructional variables. 

Gerlach (9) argued that selection of an instructional instrument (along with determina- 
tion of the teacher's instructing behavior) is based upon its ability to provide the combina- 
tions and permutations of stimulus conditions that control directly oi^ indirectly relevant 
student responses. He assumed that more than one instructional instrument will provide 
with equal effectiveness the required stimulus conditions. Thus, final selection of an 
instructional instrument may be based on what is available and what the teacher prefers 
to use. 

He noted that activities prior to media selection includej^k analysis, statement of 
objectives in behavioral terms (such as identifying, classifying^ naming, ordering, stating a 
rule, applying a rule), and development of Instructional Spedfications. 

He included in bistructional Specifications a statement of the objective, the instruc- 
tional cue, eliciting cues, and the limits (determination of stimuli to which ti\e response 
is attached and those closely related stimuli to which the response is not attached). Once 
the Instructional Specifications have been developed, he said, they control the teacher's 
instructing behavior (and selection of content, materials, media, and sequence). He noted 
that' the best possible medium for presenting the stirbulus of interest is determined by 
the stimulus-presentation characteristics of the various media. Finally, he put desired 
stimulus characteristics in a matrix with media (Table B-1), the media being subdivided 
into projected, nonprojected, recorded sound, and 3-D materials. 

The major difficulties in media selection, according to Gerlach, are in defining 
relevant responses and in specifying the stimulus conditions that are to control these 
responses. 

Gerlach and Ely (10) present a procedure in which selection of learning resources 
(performed following specification of content and objectives, assessment of entering 
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The Communication Media 



Ttitoommunication 


Sound 1 


1 Picture 1 


Line Graphic 1 


1 Print 1 


1 Motion 1 


RMording 


CImsI: AudioMotion-VitiMiMtdia 




X 


X 


X 


X 


X 


Sound film 


Television 


X 


X 


X 


X 


X 




Vkieotape 




Film TV recording 




□□□□□ 


Holographic recording 


Picturephone 


[X X X X 






Clets II: Audio^till-Vituai Madia 


Slow-scah TV 


X 


X 


X 


X 




Recorded still TV 


Time-shared TV 




X 


X 


X 


X 




Sound filmstrip 


X 


X 


X 


X 




Sound slides 


X 


X 


X 


X 




Sound-on-slide 


X 


X 


X 


X 




Sound page 


X 


X 


X 


X 




Talking book 


Class III: Audio-Semimotion Madia 


Telewriting 


X 




X 


X 


X 


Recorded telewriting 


Class IV: Motion*Visuai Media 






X 


X 


X 


X 


Silent film 


Class V: Still*Visual Madia 


Facsimile 




X 


X 


X 




Printed page 






X 


X 


X 




Filmstrip 




X 


X 


X 




Picture set 




X 


X 


X 




Microform 




X 


X 


X 




Video file 


Class VI: Audio Madia 


Telephone 


X 












Audio disc 


Radio 1 


Audio tape 


Class VII: PrintMadia 


Teletype 








X 




Punched'paper tape 



NOTE: R. Brctz: The 5«/ecf/on of Appropriate Communieetion Medie for Imtructlon: 
A Guide for Designers of Air Force Technhel Treining Progrems, 1971, OThe 
Rand Corporation, (g). Reprinted by permission. 



Figure B-1 



PrMfitition Dimmtiofis 



Stiimikit 



Rttpontt 



< 



< 



Encoding 
Environmental structure 
Pictorial 
Symbolic 
Verbal 

Reiponae Demand 
Covert 
Selective ' 
Constructed 
Vocal 
Motor 
Affective. 

Menaler Purpose 

Need Management 
Attainment Management 
Prescriptive Management 
Enrich Management 
Motivation Management 
Systems Support 



< 



< 



Duration 

Transient 

t 

Length of time the 
preMntation renoains-intact 

Persistent 



Response Demand Frequency 

Frequent 



t 



Sparse 



Instructional Management 
Frequency 

Frequent 

t ■ 

Frequency 
of decision 
to change 
presentation 

\ 

Sparse 



NOTE: DT. Totti and Ball. "A Behavioral Approach to Instructional Design and Mmlia Seltction/' OAVCom- 
munlcBthn R§¥l$w, <8). Reprinted by ptrmitilon. 



Figure 8-2 
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Mtdia CtamifM by Encoding Viniii Duration 



Dli 



Envirom^tal 



Pictorial 



Symbcilic 



Vtrbal 



Dtmonstration 



Film 
VidM 



Animation 



Convtrsation 
Ucturt 



Laboratory 
FiaM Trip 

Slida 

PI*Worlcbook 
F|^*Gard 



Flash*Card 
PI*Worl(book 



Tutor 



Group Diicuailon 



Trantiant 

Duration Di mansion 



Obiact 



Photography 

llluftratadTaxt 

Pointing 



Diagram 
Bladcboard 



Taxt 
Manual 



Persistant 



NDTE: DX Tosti aixl JM. Ball. ''A Bahavioral Approach to Instructional Dasian and Madia atlMtion/' 04^ Com* 
mufikMm HmHm, (f). Raprintad by parmliskHi. 



Figura S-3 
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Tutor 
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Tutor 



Tutor 
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Tutor 
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Table B-1 

Classification of Media According to Their Stimulus Characteristics* 



CHARACT6RISTICS 




IVoffctid 


Non* 
FiroftcMd 


Raoordtd 

Sound 




<• 
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1 

E 


«• 

M 

u 

ll 

11 


{ 

1 

t 

I 

o 


c 
m 

m 
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Si 


j| 


I 
1 
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• 


Sound Track 


t 

£ 
c 

1 


s 

\ 


• 


Community Trips 


Bfack tnd white 


X 


X 


X 


X 


X. 












X 


X 






Color 


X 


X 




X 


X 












X 


X 






Hifbmfity 


X- 


X ^ 




X 








X 


X 




X 


X 


X 


X 


SinullMNOiK pictn iMd 
ttxt 


X 


X 




X 






X 








X 










X 




X 


X 






















InttcntVy controlM 


X 














X 


X 


X 


X 








Hi|h 0OlV|IOMQt 


X 


X 


















X 




X 




Hiiiirfdoction 


X 


X 


















X 




X 




S^imtiiiiiool 
























X 


X 


X 
























X 




X 


X 


jSSafciiltananiH Mill jjlM ' 

stHoulMniidfiiank 










X 




X 


X 


X 


X 


X 








Sowid sccoiiipMivi(i| 


X 




















X 








rfiid moiim 




X 










X 


X 


X 


X 


X 








IMCiOB 


X 


X 












X 




X 










Tdki^ikMMt 
ftiinivniiiinM 


X 


X 


X 


X 


X 


X 










i( 


X 






i«VMi| Of noirvinoi 






















X 




X 




Fit xibifitv and wriition 
of rtpfMMNttioii 






X 


X 




X 


















DiffirtmM iQiMMnttNNi 
of conpoMiits or 






X 
























( ocfvoiMtic 






X 






X 


















Ftu«Nion for ovart 
raipomas 














X 








X 








Static 


X 


X 




X 


X 




X 




X 


X 


X 


X 


•X 


X 


Dynamic 






X 






X 




X 














Succmiw approxiinationt 
of tarminalbahaviar 














X 
















Individual pieini 














X 


X 
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X 


X 






X 
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X 
















Upiyndironiation 




















X 


X 
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X 


X 1 X 




X 









'y^ jSoriach, "Solmrting an Instructional Modium/' chaptor in Mgdm CompMnen for Tmchm, 
WA Mfeiarhenny (edj, (9). 

moans "Yas/''A blank moans "No/* 



behaviors, selection of approadi and techniques, along with group size and time allocations) 
is based upcm responses rather than on stimuli alone. They specified the following five 
fictors tot screening media: 

(1) Appropriateness. Is the medium suitable to accomplish the defined task? 

(2) Level of sophisticaticm. Is the medium on the correct level of understanding 
for my students? 

(3) Cost Is the cost worth the potential learning bom this particular medium? 

(4) Availabilifar . Are the material and equipment avaibble vrtien I need them? 

(5) Tedmical quali^. Is the quality of the material acceptable-readable? 
visible? audible? 

The selecticm matrix used by Gerladi and Ely is shown in Figure B^. 



MeiKt lilection Matrix 





1 

"m 
m 

OC 


Rapresentations 


Pictures 


Audio Racordirigi 


Programs 


Simulations 


Verbal 


Graphic 


Still 


Motion 


asisciioii raciofs 


















Appropriatenstt 




Thm Imtk 


xsts mun 1 


It contidi 


vad 








Level of Sophistication 




first. If no barrim ar« pTMfi 
than mow to nhiartiwi bakm 


It. 

r. Iff 








Cost 


1 bwiiiri art pmant* Mk if thay 








Availability 




canbalw 


ad on obj 


■ctivaK* 










Tachnicat Quality 




































To Mantiffy 


















To Name 


















ToDanriie 


















ToOrdar 


















ToConstnict 


















To Dhplay Ate^jdas 


















To Parform Motor Skills 



















NOTE: Vamon S. GarMi and Donald P. Ely, 7mehiti§ md Skdlh: A S^smmk Afipfotcff, 0197I. RapnntMl by 
p ar m inion of Prantiot4lall, Inc^ Enalaiwood Oifft, Nawr Jarsay (IS). 

Figure B-5 



Two additional preconditions not integrated directly into the selection matrix are 
selection of group size and the events of instruction. The Gerlach and Ely media selection 
rule requires that a medium of instruction be selected on the basis of its potential for 
implementing the behavior stated in an objective. 

Nunnelly and others (11) used a matrix (Figure B-6) to relate methods and media to 
learning tasks and suggested another matrix (Figure B-7) to relate job tasks to media, but 
they left the cells blank. The criteria tKey used for selecting training aids are given in 
Table B-2. Finally, they suggested cost-effectiveness and other criteria for selecting a 
single method-media combination. 



Optimal Allocation of Metfiod/Media to Learning Objectives 





Simulator 


Part-Task 
Trainer 


Animated 
Panels 


Mock-up 


Television 


Moving 
Pictures 


Still 

Pictures 


Recorder 


Programmed 
Instruction 


Lecture/ 
Discussion 


Learning Identifications 


Yes 


Yes 




Yes 


.'es 


Yes 


Yes 


Aural 
Only 






Learning Perceptual Discriminations 


Yes 


Yes 




Yes 


Yes 


Yes 


Yes 


Aiffai 
Only 






Understanding Princtptes and 
Relation^ips 






Yes 












Yes 


Yes 


Learning Procedural Sequences 


Yes 


Yes 






Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Making Decbions 










Y« 


Yes. 


Yes 


Yes 


Yes 


Yes 


Performing Skill Perceptuai-Motpr Acts 


Yes 


Yes 




Yes 















NOTE: Adapted from C.L. Nunnelly, er aA "The Instructtonal System Approach to Maintenance Technical Training: 
Development and Implementation Model/' Himm Facixm (11) OHumMi Factors Society. Reprinted by 
permission. 



Figure B-6 

Parker and Downs (12) base media selection on the training objectives. The two 
criteria stated for selection of media are (a) its appropriateness to'the particuW task 
performance (from which the training objective is derived)» and (b) its use to create an 
environment that best induces learning. They named five features that influence learning 
efficiency: trainee readiness, opportunity for correct response, guidance toward correct 
response, reinforcement, and motivation. 

Parker and Downs also name five characteristics of effective training: apprbpriateness 
to initial performance level of trainee, relation to training objective, repeated [xractice of 
difficult perfomumce, sample of problems of graded difficulty, and similarity to opera- 
tional tasks. They also indicated cost as a selection factcMr, and consolidated their recom- 
mendations with first, second, and third choices. The results are shown in Table B-3. 
Their media selection matrix is shown in Figure B-8* 

Wilshusen (13) classified many foctors to be considered in selectir j( media according 
io four categories: learner characteristics, task requirements, materials, and transmisuon. 
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Method/Media Selection Matrix 





Simulator 


Part-Task 
Trajner 


Animated 
Panels 


Mock-up 


Television 


Moving 
Pictures 


Still 

Pictures 


Recorder 


1 £ 
fc .9 

§6 

is 


Lecture/ 
Discussion 


Assemble-Disassemble 






















Remove-Install 






















Adjust-Align-Calibrate 






















Test-CnecK Out 






















rianoie- 1 ranspoft 






















Package-Uhpackage-Protect 






















ConnMt-DiscnnnACt 






















Check-lnspect-Monitor 






















Service^^ 






















Tioubleshoot 






















Record 






















Communicate 






















Supervise 






















Operate 






















Other (list) 























NOTE: Adapted from C.L. Nunnelly, er at 'The instructional System Approach to Meintenenoe 

Technioel Training: Development and ImplementatkMi Model/' Hummt Facton (11) O Human 
Factors Society. Reprinted by permission 

Figure B-7 



Table B-2 

Training Aid Selection Criteria and Task Specifications^ 



Level I: Requires simple identification of components,vunderstanding of discrete perceptual, motor, 
and/or perceptual-motor behavior segments. 

1. No emphasis in operational integrity. 

2. No requirement for high simulation fidelity. 

3. Low order of task complexity. 

4. Learning objective: 

a. Awareness 

b. Discrimination 

Level II: Requires learning of specif ic procec/it/res using equipment which represents operational 
configuration. Emphasis on orderly sequences, parts relationships, test, check, etc. 

1. Emphasis on feedback for test. 

2. Emphasis on positive transfer to real equipment requirements. 

3. No high simulation fidelity for internal operation of trainer to aircraft. 

4. Learning objective: , ^ 

Prerequisite a. Awareness 

b. Discrimination 

NEW c. Application within established order with self*initiated strategies 

Level III: Task specifications require learning of single sets of tasks which represent only part of total 
operational requirements. Emphasis on operational integrity for maximum transfer to real world. 

1. Emphasis on system operational identical to aircraft operation. 

2. High simulation fidelity. 

3. Emphasis on continuous feedback and system integrity for trainer. 

4. High or<Jer of task complexity: 

a. Self*initiated responses based on continuous changing of S*R components (displays, 
controls). 

b. Responding to a variety of S-R configurations, requiring immediate and unique response 
modes. 

a Real time continuous for operation model, 
d. Learning objective: 

(1) Awareness 

(2) Discrimination 

(3) Procedure Applicatiofr^ ^ 

(4) Application of analysis and decision-making commitments (problem solving 
strategies). 

Level IV: Task specifications require learning of total operational task as required for actual full opera- 
tion of aircraft. 

1. Highest simulation fidelity. 

2. Total feedback-response cycle required for aircraft operation in all normal emergency modes. 

3. Highest order of task complexity. 

4. Learning objective: Total operational proficiency for subsystem. 

a. Analysis, application, etc., through correct decisions at appropriate time, using correct 
^ procedures, and correct problem*solving strategy to achieve stated measure objective 

for aircraft. 

^C.U. Nunnelly, 0t ah 'The Instructional System Apprcath to Maintenance Technical Training: OevelopnrMnt 
and Implementatioh Model," Human Factors {uy Human Factors Society. Reprinted by permission. 
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Table B*3 



Example of Information Appearing in a 
Qualitative and Quantitative 
Personnel Requirements Information Document* 



Item 



Task Index 
Number: 



Data Sources: 

Task Title: 
Job Operation: 

Duty: 

Position: 

Work Area or 
Location: 

Task 
Description: 

a. Equipment: 

b. Indicator 
(Display): 



c. Action: 

d. AGE and System 
Equipment: 



e. Feedback 
Indication: 



Remarks 



A numbering system is used to identify each task; usually 
corresponding to the numbering in the system functional 
flow diagrams. 

For each major source of data, as an aid in later verification, 
record the date and equipment %ta^s (see MIL'D-26239A) 
at the time the information becomes available. 

The task title should be short and should tell what the 
man does in. functional terms. 

A job operation is the performance of a support or mission 
function of a system; usually performed at a singfrlOcation 
as a unit of work, i.e., it has a definite beginning and 
ending. 

The larger units of work under a job operation assigned to 
an individual in the execution of a position. Duties are 
made up of operationally similar tasks in a given operator 
or maintenance position. 

The position-type title or AFSC to which it is believed 
each task should be assigned. 

The identification of the work area or location. 

Normally, the following will be includjsd; — 
-Describe to the levels of subsystenfand component. 

-Information, perceptible to an indivkiual, concerning 
some aspect of the functioning or operational status of 
equipment; includes communication with other personnel. 
-Give a detailed breakdown of all (essential) actions to 
be perforined in response to display indications. 

-Identify the aerospace ground equipment (AGE), or 
other system equipment including special tools, and 
test equipment, used in performing the task. 

-Describe the procedures and indications that inform 

the man of the adequacy or inadequacy of his performance. 



'Continued" 



ERIC 



1^ 
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Table B-3 Continued 



Example of Information Appearing in a 
Qualitative and Quantitative 
Personnel Requirements Information Document^ 



item 



Remarks 



Type of Task: 



Frequency of 
Performance: 



Performance 
Time: 



Criticatity; 
Newness: 



Other Air Force 
Positions: 



Safety Factors: 



Personal 
Equipment: 

Skills and 
Knowledges: 



Indicate the qualities or characteristics of the task 
through the use of such categories as (a) fixed procedure, 
(b) variable procedure, (c) motor skill, (d) system analysis, 
(e) circuit analysis, etc,. 

Use 1 for once, 2 for twice, etc., in conjunction with H 
(hourly), S (per shift), D (daily), W (weekly), M (monthly). 

An estimate in hours and minutes of how long it normally 
will take Air Force personnel to perform each task, 
if appropriate; also cite the maximum permissible time 
for task. 

Indicate the effect of failure to perform the task upon 
the success or failure of the job operation or mission. 

Indicate the extent to which the task is new to the Air 
Force or the extent to which the task is partly new and 
partly old. 

Identify tasks whose performance depends upon or 
interacts with the performance of personnel in other 
position types; identify these associated positions and 
. indicate the functional interrelation of the tasks 
involved. 

Any known or presumed hazardous conditions or 
sourcesof danger, whether mechanical, electrical, or — 
chemical, should be identified and described in sufficient 
detail to provide guidance for their solution. 

Requirements for personal equipment (including 
protective clothing). 

Define the kinds of job oriented knowledges and skills 
involved in performing the task. 



^Reprinted from Parker and Downs (12). 




The Selection of Training Media in Relation to Specific Training Objectives 



ID 

s 

.E 

!e 

Training Objectives i~ 


Simulator 


Procedures Trainer 


Animated Panel 


Chart 


Training Film 


Transparencies 


Mock-Ups 


Television 


Teaching Machines 


Learning Identifications 


3 


3 




2 




1 


3 






Learning Perceptual Discriminations 


2 








1 


3 








Understanding Principles and Relationships 


1 


2 


1 


Z 


1 


2 


1 (Oper) 

2 (Non*Oper) 


1 


2 


Learning Procedural Sequences 


2 


1 






3 










Making Decisions 


3 








2 








1 


Performance Skilled Perceptual*Motor Acts 


1 


2 

















NOTE: Reprinted from Parker and Downs (12). 

Figure B-8 



A matrix (Figure B-9) is intended only to remove inappropriate media from consideration 
on the basis of four separate groups of factors. 

Decisions about learner characteristics and task requirements are necessary before 
entering the matrix. Wilshusen*s directions for using the matrix indicate that skilled 
personnel would be required to develop both learner characteristics and task requirements: 

LEARNER CHARACTERISTICS 

Large, medium , small, individual : reiet to sizes of groups of learners. 

Visual : learner characteristics dictate that the stimulus material be visual. 

Audible : learner characteristics dictate that the stimulus material be audible. 

Learner-paced : learner characteristics dictate that the rate of presentation be controlled 
by the learner,"^ 

Response : the medium contains provision for incorporating demand for learner response. 

Self-instructional : learner characteristics dictate that stimulus materials be so designed 
that learner is able to use them with little or no supervision. 

TASK REQUIREMENTS 

Motion : task requirements indicate that motion must be depicted. 

Time (exp/contract ): refers to the possibility of expanding or contracting length of 
presentation as compared with real-time experience of same phenomena (e.g,, slow 
motion or speeded motion pictures, comi»re8sed or expanded speech devices). 
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Matrix for Selecting Media 



If — ► 

1 

f 


Learner Characteristics 


Task Requirements 


Materials 


Transmission 


+ 

8 


Medium 30-100 


Small 2-30 


Individual 


Visual 


Audible 1 


Learner Paced 


Response 


Self*lnstructional 


Motion 


Time (Exp/Contract) 


Fixed Sequence 


Flexible Sequence 


Sequential Disclo. 


Repeatability 


Context Creation 


Affective Power 


Obtainability 


Reusability 


Time to Obtain 


Cost (4 Copies) 


Simplicity (Eq,) 


Availability (Eq.) 


Controlability 


Freedom from Oist. 


1 Darkening not Req. 


Real Obie<it 


'M 




















































Model of 
-fleal Object * 
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Flat Picture 
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Empty Cell * applicable 

Partial Shading « partially applicable 

Full Shading « not applicable 

NOTE: LJ. Briggs: Handbook ofProemlum for tho Dot^n of instructhn: Monograph #4, C Aniarican Institutes 
for Research, reprinted by permission. An earlier version appeared in Wilshusen (13). 

Figure B-9 
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Fixed Sequence : refers to characteristic of medium that does not permit change in 
sequence of presentation beyond forward or reverse. 

Flexible Sequence ; medium permits change in order of presentation of stimuli. 

Sequential Disclosure; medium permits revelation of material bit by bit and allows 
retention of prior bits as further bits are revealed. 

Repeatability ; medium allows complete or partial redisplay. 

Context Creation ; refers to capability of media to transport learner from awareness of 
real world to context artificially contrived. Motion pictures are an obvious example, 
but it is our contention that all media have this capability to some degree. A book has 
it, for example. 

Affective Power ; all media have the power to move people emotionally to some degree. 

MATERIALS 

The items in this group are reasonably clear. 

TRANSMISSION 

Simplicity ; How simple is the equipment to operate? 

Availability ; How readily available is the equipment required to display the stimulus 
materials? 

Controlability ; How much control over the transmission can be exercised by the 
instructor? (Start/stop, slower/faster, freeze frame, volume change, forward/reverse, 
repeat, switch to different medium*) 

Freedom from distraction ; To what extent does the equipment distract the learners 
from the intended stimuli? 

Darkening not required; medium can be presented without necessity of darkening 
learner environment. 



References for Appendix B 
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Ely, D.P. (ed.). "The Changing Role of Audiovisual Process in Education; A Definition and a 
Glossary of Related Terms,'' Audio-Visual Communication Review, vol. 11, Supplement 6 
1963, pp. 1-148. J. V 

Knowlton, J.Q. "A Conceptual Scheme for the Audiovisual Field," Bulletin of the School of 
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Lurasdaine, A.A. "Educational Technology, Programmed Learning and Instructional Science," in 
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Apptndix C 



BASIC MATHEMATICAL FORMULAS FOR 
USE IN COST ANALYSIS 



Mathema^cal formulas for use in training-cost analysis are rather simple equations - 
for calculating and comparing training output-unit costs for alternative training programs. 

Before stating the formulas, we must note that basic develdpmental costs, once the 
investments are made, do not contribute any more to the cost of training many students 
than they contribute to the cost of training a few. In other words, basic developmental 
costs can be amortized over the life of the course. Operational costs , however, remain the 
same for each group of students, regardless of the number of groups trained. In other 
words, it costs as much to operate the training system for the 15th group, for example, 
a% it does to operate the system for the first group (provided the operational costs are 
hot changed)* 

Therefore, as the number of groups of students increases, the basic cost per group 
(or per student, or per student-hour of instruction) decreases, but the operational cost 
per output unit remains constant. For these reasons, basic developmental costs can be 
thought of as variable costs and operational costs ss fixed costs. For simplicity, these 
two kinds of costs will hereafter be referred to as ^^basic" and ^^operational/' 

The mathematical formulas for calculating trainingj>utput-imit ^^osts and comparing 
them can now be stated. ^ 



FORMULAS FOR CALCULATING OUTPUT UNIT COSTS 
(UNDEPRECIATED VALUES) 



If output unit is a group of students 




+ 0 



Where: Q 



cost per group 
basic cost 
operational cost 

total number of student groups (output imits) 



B = 
O = 
N « 



Example: If B 
O 
N 



^ $45,C00 . 
= $5,000 

- 20 groups of students 



We would have: 




Cg = $2,250 + 5,000 

Cg - $7,250 (The output-unit cost of the 20th group of students) 
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If ou^ut unit ii etch student 



C .1 + 2 



Where: Cg " coft per student 
B * bade cost 
O ■ operational cost 

N ■ total number of students who have taken the course (including the 
present group) 

n ■ number of students in the present course 

Elample: If B * $45,000 
<a O « $6,000 

N " 600 (the number of students in 20 groups-the output units) 
n - 30 



Wewouldhave: C, 



600 30 
Cg - $76 + $166.67 

Cg * $241.67 (cost per student in the 20th group) 

If output unit is each student^hour of instruction 

BO 

Where: Cg.)| ■ cost per student^hour 
B * basiccost 
O * operational cost 

N ■ total number of student^hours accumulated for the course 
(including current student*hours) 

n » number of sfaident^hc^ayin^the current course 

. Example: If B - $46,000 
O - $6,000 

N » 24,000 (total number of student-houn required to train 20 
groups of students) 

n ■ 1,200 (number of hoius required to train the 20th group) 



$46,000 $5,000 
Wewouldhave: Cg.h * ||g^ + ^ 



I 



Cg.h = «n.88 + $4.17 

Cg«)| « $6.05 (cost per student*hpur for the 20th group) 



As the total number of output units, N, increases by course repetition, the basic 
cost per output ur*t, decreases so that with the accumulation of a large number of 

output units, the basic cost per output unit tends Ijward zero* So in the long tun, the 

O ' 
output- unit cost would be almost 4'qual to — , the operational cost divided by the number 

of output units associated with one group of students* These cost relationships are shown 

graphically in Figure 

Depreciated unit cost formulas will have the form; - ' \ , 



Where: n « depreciation time* Similar formulas can be developed for the different 
output units selected for cost analysis* 



FORMULA FOR CALCULATING CUMULATIVE OUTPUT-UNIT COSTS 

If training decision*makert want to know the relationships of output-unit costs in 
terms of their accumulated total costs for a number of output units, <k>st analysts ^n 
use the following formula* For sitnplicityt the formula shown is for a group of stuu^nts 
(taking a course) as the output unit* 

On « B + N • O 

Where: Cff ■ cumulative output-unit coot 

B » basic cost 
O operational cost 

N ■ total number of output units 

The resvilts of calculating cumulative output<>unit costs can be shown graphicallyt as 
in Figure &2* 

However, if the basic cotta are aubject to defffedation over m output units, the cost 
will be: 

CN-N(|*N)0-N(f*0) 
For N < m* When N > m, basic costs have been paid for, therefore: 

On - N • O 

Although the output unit used to exemplify the formula for calculating cumulative 
costs was a group of students, the cost relationships for other outpu units (cost per 
student, and cost p^r student-hour of instruction) would also be depicted as those in 
Figure C-2* 

To compare two or more training programs, the form^Uas for calculating unit costs 
and cumulative unit costs wouki be used for each program; and when the results are 
{dotted in a graph, decision makers can easily compare the programs by noting differences 
in their unit costs per number of output units* 

The value of these analyses for comparing training programs will he shown later with 
an 9 "tual example. But first, one mote useful mathematical formula most be presented* 
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Relationship of Basic Costs, Operational Costs, and Combined Costs Pier 
Ciiit tb Number of Output Units for One Training Program 



C 
3 




Number of ou. jt units 
Figure C-l 



Relationship of Cumulative Basic Costs, Operational Costs, and 
Combined Cumulative Costs Per Unit to Total Number of 
Output Units for One Training Program 



t; 
o 



3 



Combined Cumulative 
Output Cost Pe> i »iit 



Cumulative Operational Cost 
Pe- Unit 

Cumulative Basic Cost 
Per Unit 




Number of Output Units 
Figure C-2 
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A FORMULA FOR FINDING WHEN UNIT COSTS OF 
TWO PROGRAMS WILL BE EQUAL 



By equating the unit cost for two programs, solving the equation for yie lds ♦^he 
number of output units for which the unit co^t of the two programs will be equal. 

Bi Bo 



The solution for N: 



O2-O1 



Where: N = the Nth output. wnit (e.g., 4th, 5thf etc.) at which the unit costs*' 
will be equal - - 
hi - bade cost of the.first program 
B2 - basic cost of the second program 
Oi = operational cost of the first program 
\ O2 ^ operational cost of the second program 

This formula can be us^ only under certain conditions: 

(1) The output units for each course must be the same. 

(2) If tlie value of N is a whole number and a fraction, the value must be 
rounded to the next whole number. 

Both basic costs (Bi and B2) and operational costs (Oi and O2) must be unequal. If 
basic costs (Bi and B2) are equal, expenditures for the program with the larger operational 
costs will always exceed the alternative program. Similarly, if operational costs (Oi and O2) 
are equal, expenditures for the program with the larger basic cost will always exceed the 
alternative [program (depreciation not considered). 

This formula will be used in the next section with actual basic and operational costs 
for two different programs. 



AN ACTUAL EXAMPLE 



To i^ovide an illustration of the use of these cost-analysis formulas for calculating 
output-unit costs and cumulative unit costs, they were applied to data from an actual 
project designed to improve teletypewriter operator training. 

The old program required 100 hours of training, but the experimental program 
reduced the number of hours to 30. The number of students trained in each course was 30. 
There were no significant diffei-ences in student performance on the final tests, but the rate 
of learning in the experimental program, was, of course, much greater. However, this 
criterion was not sufficient as a basis for n decision as t6 which program should continue 
in use, because the basic cost of the experimental program would be much more than the 
old one, so cost analyses were performed to compare the two programs. 
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The following (Nrogram costs were accounted for: 
The experimental program 



Basic developmental cost 

Investment 20,000 (These figures are in arbitrary units. 

Salaries - 2,000 but are based on the actual project 

Tape leonrders 3,000 data.) — ; i-r,M r ^< u 

Overhead 1,000 

TOTAL 26,000 

Operational cost 

Instructors 600 
Students 900 
Maintenance 400 
Space 600 
Overhead 100 

TOTAL 2,600 

The oM progwun 

Basic developmental cost 

Investment 5,000 (Investments commcm to both pro- 

Overiiead 500 _ grams, such as for teletypewriters 

TOTAL 5 500 installation, were excluded.) 

Operational cost 

InstrucUNTs 1,000 
Students 3,000 
Maintenance 1,000 
Space 1,100 

TOTAL 6,100 



The output unit for eadi inrogram was a group of 30 studaits. 

Osculations were made for evary other replication of the courses to predict changes 
over number of presentations, and the results are shown in Table C-1. The results for unit 
costs are graphically represented in Figure C-3. 

Before the graph in Figure &3 was prepared, the following formula was used to find 
the output unit at which their unit costs would be equal: 

Where: N = the output unit at vriiidi the unit costs would be equal 

Bi - basic cost of the experimental program 

B2 - basic cost of the old program 

O2 = operational cost of the experimental program 

O2 - operational cost of the old program 
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Table C-l 



Output UnitCosts and 
Cumulative Output Uni ^ Costs for 
illustration of Altamative Training Programs 





Exptrinwmal Prognm 


CM Program 


Count 


Unit 


Cumulativ* 


Unit 


CunMlati<e 


Nufflbtr 


Cost 


Cost 


Cost 


Cost 


1 


28.600 


28.600 


11.600 


11.600 


2 


15.600 


31.200 


8350 


17.700 


4 


9.100 


36.400 


7.475 


29300 


6 


6.733 


41.600 


7.016 


42.100 


8 


5.225 


46.800 


6.788 


54300 


10 


5.200 


SZjOOO 


6.650 


66.500 


12 


4.766 


57.200 


6.558 


78,700 


14 


4.457 


62.400 


6.493 . 


90300 


16 


4.225 


67.600 


6.444 


103,100 


18 


2.044 


njaoo 


6.405 


115300 


20 


3.900 


78.000 


6375 


127.500 


22 


3.78: 


83.200 


6.350 


139.700 


24 


3.683 


88.400 


632S 


151300 


26 


3.600 


93.600 


6311 


164,100 


28 


3.528 


98.800 


6.296 


176300 


3 


3.466 


104.000 


6383 


188300 


35 


3.343 


117.000 


6357 


219.000 


40 


3.250- 


130.000 


6338 


249.500 



Thus, N = ^Ij^] 2^600 " ^ interpreted as 6. 

The graph in Figure C-3 also shows that the unit costs would be equal with six out- 
put units. It shows, too, how much less the unit costs of the experimental program would 
be when total number of output units is greater than six. In additi<»i. Figure C-3 shows 
that after eight output units, the output-unit cost of the experimental {Nrogram is less than 
the operational costs per output unit of the old program* Thus, aftar eight output units, 
unit cost-of the experimental program is less than the unit, cost of the old program could 
ever be. (Operational costs represent the lower cost limit as the number of presentations 
increase and the basic cost per output unit approaches zero.) 

Figure C-4 show^ that cumulative ou^mt-^mit costs of the two programs would also 
be approximately equal with six output unit<<, and that for output units greater tiian ax 
the cumulativa cost of the experimental program compared to the cumulative cost of the 
old ptogtam becomes increasingly lower. 

The formulas shown are sufficient for the cost analysis problems exemplified. For 
other more complex problems that may arise in Army training cost analysis, suitabk 
formulas would be derived. 
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Output Unit Costs Per Total Number of Output Units for 
illustrative Comparison of Two Training Programs 
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Appendix D 
USEFUL COST-ANALYSIS AODELS 



THE COST-ED MODEL 

The main features of the COST-ED Model may be described as follows:' 

"The COSTtED Model is a powerful analytic tool which Educ .lion 
Turnkey Systems analysts have validated and used in over 100 programs 
during the last 16 mo\ths in conducting economic analyses of instruc- 
tional programs. COST-ED Analysis provides the school administrator 
with a managerially useful description of the costs of programs, plus 
the flexible capability to make equal-cost tradeoffs for instructional 
program redesign. . . 

The following summary of the COST-ED Model is based upon a description provided by 
Educational Turnkey Systems, **The COST-ED Model, A New Economic Tool for the 
School Administrator/* 

The input data for computer analysis are provided by the client. The list of computer 
printouts that result illustrates the scope and kind of analyses performed: 

(1) Cost-Analysis Summary. Provides a one-page digest of the key results 
characterizing an instructional program. 

(2) Client Data Listing. Identifies, reviews, and validaf s all raw data used in 
constructuig the COST-ED Model. 

(3) ProgranK^ost Analysis, Shows in detail the manner and amount by which 
each economic factor contributed to the total costs, i gives the breakdown of total 
costs I: " function, by resource type, and by particular iractaristics of each resource. 

(4) Sendtivity Analysis. Shows how "sensiti^^'* the total cost value is to 
changes in each economic factor or design feature. 

(5) Economic-Factor Ranking. Focuses attention on those economic factors 
that have the greatest impact on total cost per student-year or per unit of achievement 
and shows the relative economic importance of individually managed program design 
features. 

(6) Program-Comparison Summaries . Give summary descriptions of various 
COST-ED models for use in comparison of program characteristics with those of the 
basic model. 

(7) Custom-Senritivity Analysis. Shows how changes in one economic factor 
affect requirements for any other economic factor or cost subtotal. 

(8) i^ustom-Tradeoff Analysis. Provides equal-cost alternatives for funds 
allocation between two economic factors, and shows how one economic factor, or 
corresponding program design & tture, must be changed to absorb the cost differences 
related to a change in another feature; also shows how changes in total funds available 
may be split among changes in economic factors or corresponding prpgram design features. 

'Letter from C. Blaschke, President of Educational Turnkey Systems, to Y. Riback, Human 
Resources Research OrgatJzation^ December 8, 1971. 
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THE WESTINGHOUSE MODEL 

Rhode et al (1) analyzed cost-analysis i»roblems encountered in a study done by 
Westinghouse Learning Corporation to provide the Air Police with data to be used in 
making decisions in quantity buying of multimedia for service-wide training systems. This 
level of decision making is the same as Level IV, as defined in this report, the Overall 
Army System LeveL 

Rhode classified media into 17 categories according to the degree of their (a) adapt- 
abi ^ for individualization of instruction, and (b) economy (rehtive costs)* To classify 
the ...edia according to adsq[>tability and economy, Rhode evaluated them according to their 
requirements for (a) training materials that had to be produced, and (b) additional equipment, 
facilities, supplies, and personnel expenses. 

Rhode found that attempts to establish costs of media installation and uses, apart 
.firom.|Mkrticu]ar settings and purposes, were too difficulty so system sizes and operations 
were used as delimiting parameters. It also inroved necessary to limit the number of cost 
foctors that varied in the different systems. These factors were: 

(1) Student Load. The number of students in training during ghren pt « jds of 
time. (Student loads ranged firom 200 to 1800 students.) 

(2) Hour Load. The number of hours students are engaged in training (on the 
media). (Hour loads ranged from 150 to 1350 hours.) 

(3) Duration. The duration of training i^rograms. 

(4) Course P « ji. The homogeneity of courses and the amount of course 
changes. 

(5) Locati - The location and concaitration of students. 

(6) Extended Time Frame . The length of tiie time frame serving as the focus 
of analysis (iinyears). 

Ck>st considerations were' further limited by defining initial costs as those incurred for 
purchases of equipment, fridlities, and supplies that have useful values of more than one 
year or purchases that are not made each year, and by defining operating costs as those 
incurred in purchasing goods and services every year. 

In defining these constraints on cost considerations, Rhode and his colleagues 
pointed out that for this high level of decision making it is not necessary to include all 
possible costs, but only all critical frictors that have a major impact on total costs, so 
that various media combinations for specific purposes can be compared. Thus, the results 
give ''supportable" cost approximations that can be arranged in ah order of magnitude 
for comparison, rather than complete and unimpeachable cost totals. 



THE U.S. OFFICE OF EDUCATION MODEL 

Sovergin (2, 3, 4) prepared a three-volume report for the U.S« Office of Education 
pertaining to cost analyses of large-scafe media installations for use by public school 
systems. In the first volume, Sovergin defines a hypothetical educational task as a basis 
for comparing costs of alternative media systems, and concludes that: 

(1) Objective cost estimates for pla^ming and operating media systems should 
be made after assessment of school environment factors and instructional factors (e.g., 
learning objectives and types of presentations). 
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(2) Specification of appropriate sensory stimuli and designs of alternative media 



systems must precede cost estimates for media production, distribution, and (TV and radio) 
reception. 

In the second volume, Sovergin uses an instructional task and a hypothetical educa- 
tional environment as the basis for analyzing media costs, and defines what he considers 
a feasible cost structure as a three-dimensional matrix. Each of three vectors defines cost 
sources as (a) media^production, distribution, and (TV-radio) reception; (b) capital 
requirements for initial investments and annual operating costs; and (c) environmental 
functions—regional, state, metropolitan, city, and local. 

Sovergin also jestimatesiumual per-student costs fc^ Instructional Television Fixed 
Service, airborne, satellite, UHF, and closed-circuit television, videotape recordings, film, 
radio, language laboratories, and computer dial*access systems. 

In -an appendix, Sovei^in presents analyses of component and operating costs for 
five media systems: instructional television, audiovisual systems, educational radio, learn- 
ing laboratories, and computer dial-access. 

In the third volume, Sovergin presents the minimum costs of performing three 
specific instructional tasks and a detailed cost analysis of a oomputer*assisted instructional 
system, and stresses the importance of clear recognition of alternatives and cbst-savuig 
techniques as two essentials for cost analysis. 

THE SPlAGCi MODEL 

Speagle (^ i»resents estimated annual costs of installing and operating three major 
instructional media systems: television, computer access, and computer-assisted instruction. 

To make these estimates, Speagle spedh^ i a hypothetical school district of 100,000 
students and analyzed the costs of various media configurations that might be used by the 
schools. He demonstrated ways in which ei^t key vanabl^(which include cost 
sources) may be manipulated to reduce the costs of using instructional media. The 
eight variables are: 



(5) Machinery for evaluating the quality and effectiveness of teaching 
techniques and materials 

(6) Levels and types of desired teaching programs 

(7) Rsive of learning under innovative techniques 

(8) Possibilities of replacing traditional teaching with instructional technology 



Speagle predicts that the question of whether instructional technology should supple- 
ment or replace some of the variables will be answered by comparing costs per student-hour. 



(1) 
(2) 
(3) 
(4) 



Heavy overhead and fixed costs 
Cost-saving technologies 
Geographical concentration of students 
Cooperation among school districts and schools 
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Appendix E 

A TRAINING COST-EFFECT iVENESS INDEX FORMULA 

This fomula is for use with classification of students according to test-performance 
categories as training payoff data. The first equation is ah adaptation of one presented by 
Edwards (1). - " ; 

n 

EU = X (Pn) (^n) 
n=l 

Where: EU = expected utility value 

Pn = the probability that an individual student wiU be in the 
nth category 

Uji = the utility value of the nth cat^ory 

n = the number of the nth category 

M = the total number of cat^ories 

The Pn values (probability values) axe the same as the fractions (proportions of students) 
that fall in each of the categories, which will be given by: 

Wh|»re: p^ - the probability that an individual student will be in the nth category 

Nji = the number of students in the nth cat^ory 

N = the total number of students in a course 
Combining this expression with the cymbol for the utility values of cat^ories, gives: 

EU = 27 — 
n N 

Wh^e: EU = the expected utility value „^ 

Ujj = the utility value of the nth category 

Nn = the number of students in the nth category 

N * the total nimiber of students in a course 

The cost-effectiveness index of a course can now be defined as: 

EU JfCNn/NXUn) 

CE = — = 

C C 
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The cost of training per unit of instruction when the unit of ihstructionos a group 
of students in a course was defined in Part II» Chapter 5» as; 



Thus» by combining the above two terms we can define the cost>effectiveness index 
per group of students trained in a course: 



^^V B/N + 



B/N + O 



Where: CEg = the cost-effectiveness index per group of students trained in 
a course 

Nn - the number of students in the nth category 

U|| - the utility value of the nth category 

B = basic developmental cost of the course 

O - operational cost of the course 

N - the number of students in a course 

: If EU» B> and O val;;es are held constant^ the cost-effectiveness index per unit of 
instruction » CEg> increases as the number of student groups trained increases^ because the 
unitTosts decrease over tir^e as the course is replicated^ and the unit cost eventually 
,becomes almost the same as the cost qfjoperating th "ourse^ O. Theoretically » then» the 
cost-effectivene'^s index is ultimatelyiimited by the ational cost. The theoretical limit 
can be expressed as: 

EU 

CE = limit CE = — 
O 

Where: CE = the cost-effectiveness index 

EU = the expected utility value 

O = operational cost of the course 

The use of thp cost-effectiveness index for comparison of two courses is demonstrated 
in the following section with actual data. One example shows how the index would be 
used to compare two programs when both of them are new» while the second example 
shows how the index would be used to compare an old program and a new one. 



COMPARING TWO NEW COURSES 

Two programs for training teletypewriter operators were compared by calculating 
cost-effectiveness indexes for both courses projected over time. One course (experimental) 
trained 18 males and 12 females. The other course (control) trained 19 maler>. 

Teletypewiiting tests at the end of both courses yielded the mean typing-rate 
scores shown in Table E-l» together with the standard deviations. No significant difference 
was foa.id between the groups. 
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Table E l 

Teletypewriter Mea;i Scores and 
Standard Deviations 



Course 


Mean 
Typing 
Rate 


Standard 
Deviation 


Experimental 






Males 


15.6 


3.36 


Females 


19.3 


2.«9 


Males and Females 


17.5 


3.00 


Control 






Males 


15.3 


4.60 



The expected utility values for each course were calculated by using Table E-2. 
Table E-3 shows the cost-effectiveness indexes for both courses for every other course 
replication up to 40 groups of students. The cost-effectiveness indexes in Table E-3 are 
plotted graphically in Figure E-1, which shows that the experimental program would be 
more cost effective after four courses. 



Table E-2 



Calculation of Expected Utility Values 





Experiinental Course 


Contr-jl Course 




Nn 


Nn 

IT 'Pn 


Pn"n 


Nn 


N "n 


Pn"n 


-9 
10.5 
12.5 
14.5 
16.5 
18J 
20 


0 
3 
2 
2 
4 
5 
2 


0 

.17 
.11 
.11 
.22 
.28 
.11 


0 

1.785 
1.375 
1.595 
3.630 
5.180 
2.200 


1 
3 
3 
5 
3 
1 
3 


.0526 
.1578 
.1578 
.2630 
.1578 
.0526 
.1578 


.450 
i:680 
2.000 
3.770 
2.640 

.925 
3.200 


Total 


18 




15.765 


19 




14.665 



*The Up values are the mean typ.ng rates for each of the seven intervals (categories of 
scores). 
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TabI* E-3 



Cost-Efftctiventtt Indtxtt 



4- 





Cott EffactWtntM 




Exptrimtntal 


Control 




Count 


Court* 


1 


.551 


1.264 


2 


1.010 


1.657 


■ 4 


. 1.946 


1.962 


6 


2.274 


2.090 


8 


3.017 


2.160 


10 


3.032 


2.205" 


12 


3.308 


2.236 


14 


3.537 


2.258 


16 


3.731 


2.275 


18 


3.808 


2.290 


20 


4.042 


2.300 


22 


4.170 


2.309 


24 


4.280 


2.317 


26 


' 4.379 


2.324 


28 


4.468 


2.329 


30 


4.548 


2.334 


35 


- 4.716 


2.344 


40 


4.851 


2.351 



COMPARING AN OLD COURSE AND A NEW ONE 

The data obtained in the two courses just analyzed are used again to illustrate what 
is involved in using the cos^effectiveness index to compare an old program and a new one. 

Note in Table E-4 that the number of students has been increased to 30 for each 
course. This was done because 30 students at a time would be trained in the course to be 
adopted. Therefore, the exnected utility values were converted to match the increased 
number of students. The results are in Table E-4. 

Also for this illustration, the control course was considered c . the old course, and 
the experimental course is the new course. For these calculations, it was assuzped that, 
over time, the old course had demonstrated student mean scores of 15 with a standard 
deviation of 4.5 (to make calculations easier). For the new program the scoies and 
standard deviation for the combined male and female group were used; mean score was 
17.5 with a standard deviation of 3.00. 

The new expected utility values derived in Table E-4 were used to calatlate the cost- 
effectiveness indexes for both courses. The results are in Table E-5. 

The cost«effectiveness inriexes are shown graphically in Figure E-2, and it is evident 
from the graph that the new program will be more cost-effective Uian.the old one after 
six courses. The intersection of the two curves on the graph indicates the point in time at 
which both courses would have the same cost«effectiveness index. 
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Cost-Effectiveness, lixiexes for Ccmparing Two New Courses 
(Experimental course operational cost is not shown) 



Experimental course cost*eff ectivenoK indexes ^ . 



V 

/ 

/ 

/ 



/ Control course op«ratk>rMl cost . 

■ \ 



/ 



/ 



^ Control course oost-effectSvenoss indexes 

r 

/! 

I! 

a 
I i 

7 
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10 15 20 25 30 

Number of Output Units 

Figure 



n 



-0 
10J 
1Z5 
US 
MS 

las 

20- 
Total 



Table E-4 

(Mcubrtion of Expoctad Utility Vahm 



NawCowm 



0 
3 
2 
4 

S 
7 
9 
30 



0 

.100 
.067 

.106 
.233 
.300 



0 

1.060 
875 
1J86. 
Z806 
4.256 
6iOiDO 
16.870 



OW Count 



X-|SD . 
X-SD •= 



X+^SD 
X + SO 

x+fso 



7.5 
10.00 
1Z5 
15.00 
17.5 
20.0 
22.5 



8.75 


, .1587- 


1.39. 


11JZ5 


.1498 


1.68 


ia75 


.1915 


Z63 


16.25 


—.4916 


• 3.11 


18.75 


.1498 


2St , 


21.25 


.1587 


. 3J7 



14.99 



4^ 



Cdtt*EffiKtivints$ indms 













; IWVV 








* :Oow 


Q 












1 


.590 


1.292' 




2 ^ 


1.081 


1.694 




4- 


1.854' 


Z005 




6 


Z438 


Z136 \ 




. 8 


a229 


Z208 




10 


•3.244 


2.04 




12 


asw 


2.286 " 




14 


a786 


2.309 




16 


a993 


Z327 




18 


4.172 ' 


Z340 




20 


4.326 


.Z351 V 




22 


4.462 


Z361 




24 


4.580 


Z368 




26 


4.686 


Z375 




28 


4.782 


Z380 




30 


4J67 


Z386 




36 


5.046. 


: Z396 




'40 


5.191 


Z403 





eric; 
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Cost-Effactivenes$ indexes for Comparing a New Course and an Old One 

(Experimentai course operational cost is not shown) 
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